feat(scripts): initial commit ingestor #4

Merged

puer-robustus merged 21 commits from ingestor into main 2026-08-22 23:35:37 +02:00

Conversation 2 Commits 21 Files changed 1262 +17645 -22

puer-robustus commented 2026-07-06 20:19:01 +02:00

Owner

Copy link

No description provided.

puer-robustus added 1 commit 2026-07-06 20:19:02 +02:00

feat(scripts): initial commit ingestor 1d11f24375

puer-robustus commented 2026-07-06 20:23:03 +02:00

Author

Owner

Copy link

@hernani @ijohn For your review.

Running the script (python3 scripts/ingest.py with the necessary python dependencies installed) will create a whole bunch of json files in data/projects which look like this:

{
    "name": "crypto.signText",
    "website": [
        "https://redwax.eu/rst"
    ],
    "description": "Cryptographic signatures brought back to the browser",
    "ngiFund": [
        "ngi-assure"
    ]
}

Now, this in and off itself is not much, as it is just reformatting the ngi grant level data. However, the script has the capabilities to aggregate project level data from multiple grants, IF the variables GRANTID2PROJECT and PROJECT2DESCRIPTION are correctly filled in. The question now is, how do we fill in these data programmatically? @ijohn Is there a way to get the hex data in a more consumable form than parsing the html?

@hernani @ijohn For your review. Running the script (`python3 scripts/ingest.py` with the necessary python dependencies installed) will create a whole bunch of json files in `data/projects` which look like this: ``` { "name": "crypto.signText", "website": [ "https://redwax.eu/rst" ], "description": "Cryptographic signatures brought back to the browser", "ngiFund": [ "ngi-assure" ] } ``` Now, this in and off itself is not much, as it is just reformatting the ngi grant level data. However, the script has the capabilities to aggregate project level data from multiple grants, **IF** the variables `GRANTID2PROJECT` and `PROJECT2DESCRIPTION` are correctly filled in. The question now is, how do we fill in these data programmatically? @ijohn Is there a way to get the hex data in a more consumable form than parsing the html?

puer-robustus added 1 commit 2026-07-06 20:24:41 +02:00

style(scripts/ingest.py): add REUSE header 0fbfd1f961

puer-robustus commented 2026-07-06 20:26:24 +02:00

Author

Owner

Copy link

The resulting json files are also not schema compliant yet.

The resulting json files are also not schema compliant yet.

puer-robustus force-pushed ingestor from 0fbfd1f961 to a6569922d6 2026-07-06 20:29:26 +02:00 Compare

puer-robustus force-pushed ingestor from a6569922d6 to e503946643 2026-07-10 14:10:12 +02:00 Compare

puer-robustus force-pushed ingestor from e503946643 to 024a0492af 2026-08-17 16:45:52 +02:00 Compare

puer-robustus added 15 commits 2026-08-22 23:34:13 +02:00

scripts/ingest: refactor project to grant mapping for humans ... 5afac91e50

Convert easier to edit PROJECT2GRANT dict programmatically into more
performant GRANTID2PROJECT dict.

scripts/ingest: extend project 2 grant map e637169b9c

scripts/ingest: ensure that no grant is mapped twice 91e96e1d61

scripts/ingest: give Project a .to_json() method 014f5e28b1

scripts/ingest: store property clashes on project level ... a07e941ccc

Be aware that the clashes collection is only filled up if the relevant
properties have been accessed before!

scripts/ingest: introduce (basic) merge logic for NGI json load ... 686b11ee60

Since NLNet will continue to hand out grants (even if not under the NGI
umbrella), it would be useful to be able to repeatedly import the NGI
provided json file which outlines the given grants ("projects" in _their_
lingo). At the same time, we don't want the data we have
updated/summarized/corrected for the (what we call) project level to be
overwritten by upstream.

Therefore, the current solution is to merge the existing project level json
with the project level jsons we build from the new NGI json grant load. For
projects with a single grant this means that the upstream data will be
authoritative. For multi-grant projects, those attributes where clashes have
been resolved, they stay resolved.

scripts/ingest: collect NLNet pages from grants in project bd529dbaa4

scripts/ingest: lowercase urls d6757d719b

data/projects: update data as loaded with running the ingestor script ... bbe4fdc267

In the process, we might lose some good "short_description", "website" and
"source_code" attribute data for multi-grant projects, but I don't see a way
to reliably and automatically identify the grant which provides the good
data points.

fix newlines 2cd650a58d

scripts/ingest: remove grants which have been mapped to projects ... 30301e9380

This immediately removes 255 files from data/projects.

scripts/ingest: add newline to end of json files ... 3466479774

Otherwise, the end-of-file diffs which are fixed up by the pre-commit hooks
would always appear in the git diff after a new load.

schema: TEMPORARILY remove the isFree -> provide source requirement ... 6f2de61b4d

Too many project level data currently don't fulfill this requirement. We
will have to manually add them if they are not in the upstream NGI JSON or
extract them if they are.

script/ingest: skip another category in additionalInfo description list ... 0090bbccc1

The script fix improves some project data we load.

schema: allow "source" attribute to use either https or git URL d88df16530

puer-robustus changed title from WIP: feat(scripts): initial commit ingestor to feat(scripts): initial commit ingestor 2026-08-22 23:35:05 +02:00

puer-robustus merged commit d88df16530 into main 2026-08-22 23:35:37 +02:00

puer-robustus deleted branch ingestor 2026-08-22 23:35:38 +02:00

puer-robustus referenced this pull request 2026-08-22 23:38:12 +02:00

check for integration of NGI-0 provided project json #2

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

app

  

data

  

schema

No labels

app

data

schema

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

ISOC-CH/tech-dossier-project-data!4

No description provided.