karakeep-app/karakeep

Deduplication doesn't take into account the trailing "/"

Open

#864 opened on Jan 12, 2025

View on GitHub
 (1 comment) (3 reactions) (0 assignees)TypeScript (1,356 forks)auto 404
feature requestgood first issuepri/mediumstatus/approved

Repository metrics

Stars
 (27,587 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the Bug

I am not a programmer so I may be well off base here but this is what i've found.

I have imported all my bookmarks from the various services i've used over the years totalling 12844 bookmarks. While scanning the list in Hoarder I came across some duplicates, the only difference between urls being a trailing "/". I've seen that you do deduplication on import but treat duplicate URLs with different protocols (http and https) as unique.

Long story short, I knocked up a python script with chatgpt to take my exported bookmarks and ignore the protocol and normalise the urls (ignoring the trailing "/") and the output was 2257!

Steps to Reproduce

Import bookmarks with duplicate urls but with one having trailing slashes added.

Expected Behaviour

One copy of a particular url.

Screenshots or Additional Context

No response

Device Details

No response

Exact Hoarder Version

v0.21.0

Have you checked the troubleshooting guide?

  • I have checked the troubleshooting guide and I haven't found a solution to my problem

Contributor guide

Research direction
Implement URL normalization in deduplication logic to ignore trailing slashes. Check the codebase for the deduplication function, likely in the backend. Add a step to normalize URLs by removing trailing slash before comparing.
Tech stack
typescript
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
Git
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
90