-
Introduction
As an enthusiast who loves reading and collecting online resources, I often face the challenge of managing a large number of URLs. I used to use TickTick to manage these URLs, but over time, some links may become invalid - a common occurrence on the Internet. I’ve known about the Linkwarden project for a long time, but it wasn’t mature back then. Recently, I revisited this project and found that it has matured enough to support browser extensions, iOS shortcut command import of URLs, and PWA. It can almost be said to be a perfect solution.
Setting up PostgreSQL
Linkwarden uses PostgreSQL for database storage. I usually deploy it in a Kubernetes environment. Here is an example of a PostgreSQL Kubernetes configuration:
First is sts.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql
namespace: app
spec:
selector:
matchLabels:
app: postgresql
serviceName: postgresql
replicas: 1
template:
metadata:
labels:
app: postgresql
spec:
hostNetwork: true
containers:
- name: postgresql
image: postgres:14.11-alpine3.19
ports:
- containerPort: 5432
name: postgresql
env:
- name: POSTGRES_PASSWORD
value: 'xxxxxx'
- name: POSTGRES_HOST_AUTH_METHOD
value: trust
volumeMounts:
- name: postgresql-data
mountPath: /var/lib/postgresql/data
- name: timezone
mountPath: /etc/localtime
readOnly: true
volumes:
- name: timezone
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
volumeClaimTemplates:
- metadata:
name: postgresql-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 200Gi
Next is svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: postgresql
namespace: app
spec:
selector:
app: postgresql
ports:
- port: 5432
targetPort: 5432
Setting up Linkwarden
Next is an example of a Linkwarden Kubernetes configuration:
First is sts.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: linkwarden
namespace: app
spec:
selector:
matchLabels:
app: linkwarden
serviceName: linkwarden
replicas: 1
template:
metadata:
labels:
app: linkwarden
spec:
containers:
- name: linkwarden
image: ghcr.io/linkwarden/linkwarden:v2.5.1
ports:
- containerPort: 3000
name: linkwarden
env:
- name: NEXTAUTH_SECRET
value: 'xxxxxxxxxxxxxxx'
- name: NEXTAUTH_URL
value: https://linkwarden.your-domain.com/api/v1/auth
- name: POSTGRES_PASSWORD
value: 'xxxxxxxxxxxxxxxx'
- name: DATABASE_URL
value: postgresql://postgres:${POSTGRES_PASSWORD}@postgresql:5432/linkwarden
volumeMounts:
- name: linkwarden-data
mountPath: /data/data
- name: timezone
mountPath: /etc/localtime
readOnly: true
volumes:
- name: timezone
hostPath:
path: /usr/share/zoneinfo/Asia/Shanghai
volumeClaimTemplates:
- metadata:
name: linkwarden-data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 200Gi
Explanation of environment variables:
NEXTAUTH_SECRET: A random string, those who know me know that I would randomly create a folder and then get its MD5 as this random string.NEXTAUTH_URL: The access URL of Linkwarden.POSTGRES_PASSWORD: The password of the PostgreSQL database.DATABASE_URL: The database connection address.
Other environment variables can be referred to the official Linkwarden documentation:
https://docs.linkwarden.app/self-hosting/environment-variables
Linkwarden also supports various third-party login methods, which can be referred to:
https://docs.linkwarden.app/self-hosting/sso-oauth
Next is svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: linkwarden
namespace: app
spec:
selector:
app: linkwarden
ports:
- port: 3000
targetPort: 3000
Finally, ingress.yaml:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: linkwarden-ingress
namespace: app
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
rules:
- host: "your-domain.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: linkwarden
port:
number: 3000
tls:
- hosts:
- your-domain.com
secretName: your-tls-secret
Exporting Data from TickTick
Since the number of URLs I’ve collected is close to 800, it’s obviously impractical to add them manually. Therefore, I need to first get historical data from TickTick. TickTick does not provide a data export function, but by analyzing network requests, I found that the following URL can be used to obtain all data:
https://api.ticktick.com/api/v2/batch/check/0
The returned data is in JSON format, where the projectProfiles key contains detailed information about the list, and the syncTaskBean key contains detailed information about the tasks. If you want to get all the data under a specific list, just convert the JSON data in syncTaskBean to CSV format, and then use Excel to filter the projectId field. The id in projectProfiles is the projectId in syncTaskBean.
After filtering, get the content of all title and content fields, and use a URL extraction tool (such as: https://www.67tool.com/text/extract-urls) to get all URLs.
Importing Data into Linkwarden
Next is the step of importing data. I wrote a Python script to complete this task. First, create a urls.txt file, and then write all the URLs obtained earlier line by line. Log in to Linkwarden, get your cookie after logging in, modify the cookie and corresponding Linkwarden address in the script below, and then run the script.
import requests
import json
# Read the URL in the text file
with open('urls.txt', 'r') as file:
urls = file.readlines()
# Set request header
headers = {
'authority': 'your-linkwarden-domain',
'accept': '*/*',
'accept-language': 'zh-CN,zh;q=0.9,en;q=0.8',
'content-type': 'application/json',
'cookie': 'your-cookie', # Fill in your cookie
'origin': 'https://your-linkwarden-domain',
'referer': 'https://your-linkwarden-domain/dashboard',
'sec-ch-ua': '"Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"macOS"',
'sec-fetch-dest': 'empty',
'sec-fetch-mode': 'cors',
'sec-fetch-site': 'same-origin',
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36'
}
# Traverse URL, send POST request
for url in urls:
url = url.strip()
data = {
"name": "",
"url": url, # Imported url
"description": "",
"type": "url",
"tags": [],
"preview": "",
"image": "",
"pdf": "",
"readable": "",
"textContent": "",
"collection": {
"id": "your-collection-id", # This is your linkwarden's collection id
"name": "your-collection-name", # This is your linkwarden's collection name
"ownerId": "your-user-id" # This is your linkwarden's user id
}
}
response = requests.post('https://your-linkwarden-domain/api/v1/links', headers=headers, data=json.dumps(data))
# Check response status
if response.status_code == 200:
print(f'Successfully posted URL: {url}')
else:
print(f'Failed to post URL: {url}. Status code: {response.status_code}')
Be sure to replace your-linkwarden-domain, your-cookie, your-collection-id, your-user-id, and your-collection-name in the script with your actual values.
In this way, all URLs have been successfully imported into Linkwarden.
Feel free to follow my blog at www.bboy.app
Have Fun