Commit Graph

53 Commits

Author SHA1 Message Date
Wyatt Verchere 235f4f10ef Chunking searches (#787)
* new attempt

* revised searching CTEs

* prepare fix

* fix tests

* fixes

* restructured project_item to use queries

* search changes! fmt clippy prepare

* small changes
2023-12-08 23:14:17 -07:00
Wyatt Verchere b2be4a7d67 Search overhaul (#771)
* started work; switching context

* working!

* fmt clippy prepare

* fixes

* fixes

* revs

* merge fixes

* changed comments

* merge issues
2023-12-03 07:27:12 -07:00
Wyatt Verchere a70df067bc Misc v3 linear tasks (#767)
* v3_reroute 404 error

* hash change

* fixed issue with error conversion

* added new model confirmation tests
+ title name change

* renaming, fields

* owner; test changes

* clippy prepare

* fmt

* merge fixes

* clippy

* working merge

* revs

* merge fixes
2023-12-01 20:15:00 -07:00
Wyatt Verchere 58093a9438 Modifies sql queries to use CTEs (#773)
* fixes huge slowodwn on version item

* changes!

* fixes, touch ups, indices

* clippy prepare
2023-11-30 11:10:56 -08:00
Wyatt Verchere 74973e73e6 Tests 3 restructure (#754)
* moved files

* moved files

* initial v3 additions

* moves req data

* tests passing, restructuring, remove v2

* fmt; clippy; prepare

* merge conflicts + issues

* merge conflict, fmt, clippy, prepare

* revs

* fixed failing test

* fixed tests
2023-11-16 11:36:03 -07:00
Wyatt Verchere f4880d0519 more games information, games route (#756)
* more games information, games route

* adds banner url
2023-11-14 11:01:31 -07:00
Wyatt Verchere ae1c5342f2 Search test + v3 (#731)
* search patch for accurate loader/gv filtering

* backup

* basic search test

* finished test

* incomplete commit; backing up

* Working multipat reroute backup

* working rough draft v3

* most tests passing

* works

* search v2 conversion

* added some tags.rs v2 conversions

* Worked through warnings, unwraps, prints

* refactors

* new search test

* version files changes fixes

* redesign to revs

* removed old caches

* removed games

* fmt clippy

* merge conflicts

* fmt, prepare

* moved v2 routes over to v3

* fixes; tests passing

* project type changes

* moved files over

* fmt, clippy, prepare, etc

* loaders to loader_fields, added tests

* fmt, clippy, prepare

* fixed sorting bug

* reversed back- wrong order for consistency

* fmt; clippy; prepare

---------

Co-authored-by: Jai A <jaiagr+gpg@pm.me>
2023-11-11 16:40:10 -08:00
Geometrically e9f5bd4ac1 Fix redis pool timeout (#669)
* Fix redis pool timeout

* remove search dep project issues

* run fmt + prep
2023-08-06 15:34:03 -07:00
Geometrically ea043517c5 Fix version file visibility (#630)
* Fix version file visibility

* add missing

* update prepare
2023-06-08 20:25:03 -07:00
Geometrically 59f24df294 Add dependencies to search (#578)
* Add dependencies to search

* add attrs for faceting

* run prepare

* Add user data route from token

* update to 24hrs

* Fix report bugs
2023-04-20 16:38:30 -07:00
Magnus Jensen 7ec518b41c update index to use approval date if set (#540)
Co-authored-by: MagnusHJensen <magnus.holm.jensen@lego.dk>
Co-authored-by: Geometrically <18202329+Geometrically@users.noreply.github.com>
2023-02-15 10:20:34 -07:00
Geometrically 34af33607b Update file restrictions, download counting, project colors, etc (#515)
* Update file restrictions, download counting, project colors, etc

* Run fmt + clippy + prepare
2022-12-30 13:56:41 -07:00
Geometrically 5bb188a822 Project Colors (#512)
* Inital tests

* Finish project colors

* Run fmt + clippy + prepare

* Fix dp+rp fmting
2022-12-29 17:20:50 -07:00
Geometrically 4da1871567 Public Webhook Fixes (#493)
* Public discord webhook

* Switch to jsonb for most queries + make gallery featured first

* Run fmt + clippy + prepare
2022-12-07 09:56:53 -07:00
Geometrically e96d23cc3f Initial work on new status sys + scheduling releases (#489)
* Initial work on new status sys + scheduling releases

* Finish project statuses + begin work on version statuses

* Finish version statuses

* Regenerate prepare

* Run fmt + clippy
2022-12-06 09:14:52 -07:00
triphora 820519b4f7 Move to SPDX licenses (#449)
* Move to SPDX licenses

Found a way to do this without breaking API compat, so here it is, instead of waiting for v3

Resolves MOD-129
Resolves #396

* License URL updates

* what was I thinking

* Do a thing

* Add open source filter

* Remove dead imports

* Borrow

* Update 20220910132835_spdx-licenses.sql

* Add license text route

* Update migration

* Address comments
2022-11-29 21:53:24 -07:00
Geometrically 9c7b34d5e6 Apply additional categories fix to search indexing (#428)
* Apply additional categories fix to search indexing

* fix edit version validator
2022-08-20 22:34:00 -07:00
Geometrically 992de7d66e Fix search not working (#408) 2022-07-31 15:26:25 -07:00
Geometrically b04bced37f More project data (#406)
* More project data

* Array_agg fixes + cleanup

* fix prepare

* Add approval dates to search

* Update migrations/20220725204351_more-project-data.sql

Co-authored-by: wafflecoffee <emmaffle@modrinth.com>

* Add category labels + display categories

Co-authored-by: wafflecoffee <emmaffle@modrinth.com>
2022-07-31 13:29:20 -07:00
Emma Cypress 932b0ccf24 Allow archived projects to show in search (#381)
Resolves MOD-96
2022-06-24 14:26:18 -07:00
Geometrically d29d910ac6 Add mod lists for modpacks, liteloader support, update actix, fix moderation webhook (#357) 2022-05-26 10:08:19 -07:00
Geometrically 80e00a80d5 Switch to time crate, add file sizes (#329)
* Switch to time crate, add file sizes

* Update deps, adjust pack format

* Run formatter, fix clippy
2022-03-29 19:35:09 -07:00
Geometrically 459e36c027 Run fmt, fix dep route (#312) 2022-02-27 21:44:00 -07:00
Geometrically 6740124364 Fix search returning incorrect ownership information (#261) 2021-11-24 13:20:25 -07:00
Leo Chen 13187de97d Rustic cleanups, dedups and making the code less hard to read in general (#251)
* typos :help_me:

* (part 1/?) massive cleanup to make the code more Rust-ic and cut down heap allocations.

* (part 2/?) massive cleanup to make the code more Rust-ic and cut down heap allocations.

* (part 3/?) cut down some pretty major heap allocations here - more Bytes and BytesMuts, less Vec<u8>s

also I don't really understand why you need to `to_vec` when you don't really use it again afterwards

* (part 4/?) deduplicate error handling in backblaze logic

* (part 5/?) fixes, cleanups, refactors, and reformatting

* (part 6/?) cleanups and refactors

* remove loads of `as_str` in types that already are `Display`

* Revert "remove loads of `as_str` in types that already are `Display`"

This reverts commit 4f974310cfb167ceba03001d81388db4f0fbb509.

* reformat and move routes util to the util module

* use streams

* Run prepare + formatting issues

Co-authored-by: Jai A <jaiagr+gpg@pm.me>
Co-authored-by: Geometrically <18202329+Geometrically@users.noreply.github.com>
2021-10-11 20:26:59 -07:00
Geometrically c87e72e08e Switch to alternate query strategy which simplifies code (#244) 2021-09-01 06:04:38 -07:00
Geometrically efb82847cb Switch to ARRAY_AGG for database aggregations to improve peformance + fix gallery images not showing up (#242) 2021-08-31 15:29:51 -07:00
Geometrically 76b62eda3a Allow gallery featuring, add gallery images to search, rename rejection reasons, transfer ownership route (#226) 2021-07-27 16:50:07 -07:00
Geometrically 2a4caa856e More APIv2 Fixes (#210)
* Refactor search to not spam the database with queries, new utility routes for V2

* Run prepare

* More V2 Fixes

* Run prepare + formatter
2021-06-05 20:59:21 -07:00
Geometrically 157962e42a Improve peformance of search indexing, v2 fixes + new routes (#205)
* Refactor search to not spam the database with queries, new utility routes for V2

* Run prepare
2021-06-02 18:33:11 -07:00
Geometrically 16db28060c Project Types, Code Cleanup, and Rename Mods -> Projects (#192)
* Initial work for modpacks and project types

* Code cleanup, fix some issues

* Username route getting, remove pointless tests

* Base validator types + fixes

* Fix strange IML generation

* Multiple hash requests for version files

* Fix docker build (hopefully)

* Legacy routes

* Finish validator architecture

* Update rust version in dockerfile

* Added caching and fixed typo (#203)

* Added caching and fixed typo

* Fixed clippy error

* Removed log for cache

* Add final validators, fix how loaders are handled and add icons to tags

* Fix search module

* Fix parts of legacy API not working

Co-authored-by: Redblueflame <contact@redblueflame.com>
2021-05-30 15:02:07 -07:00
Jai A e596a8f731 Fix search 2021-03-07 18:44:24 -07:00
Jai A 5b0cc73792 Change server side and client side fields to optionals 2021-03-07 17:52:04 -07:00
Jai A 2163d4465f Fix license field not working: 2021-03-07 17:07:32 -07:00
Geometrically 0ccb6cb873 Follows (#172)
* Follows initial

* Fix #171, Fix #170, Fix #169, Fix #164

* More work on follows

* Fix compile error

* Upgrade meili version, add follows to search
2021-03-04 20:35:23 -07:00
Geometrically 174dbb5e74 Fix auth params, Add params to indexing, Order version game versions, Remove version moderation, Return donation platforms in get routes 2021-01-18 10:10:45 -07:00
Geometrically f0b73fd696 Change index interval, add slug to search documents (#110)
* Change index interval, add slug to search documents

* Allow the removal of '@' for slug get

* Fix

* Remove name and rename side type

* Run prepare
2020-12-13 18:10:58 -07:00
Geometrically 0500994def Moderation + Mod Editing (#101)
* Moderation + Mod Editing WIP

* Run prepare, fix perms

* Make it compile

* Finish moderation and edit routes

* More fixes

* Use better queries

* Final Fixes
2020-11-15 19:58:11 -07:00
Aeledfyr da911bfeb8 Minor fixes to orderings and permission serialization (#102)
* Fix latest_version in search results

* Handle users with invalid permissions instead of skipping them

* Specify order of some queries, fix serialization of permissions

* Run sqlx prepare
2020-11-10 09:27:36 -07:00
Aeledfyr c8e58a1e5b Fix indexing, upgrade MeiliSearch sdk (#100)
* Hotfixes for indexing

* Handles missing INDEX_CACHE_PATH environment variable
* Exits on startup if environment variables are missing. The flag
  --allow-missing-vars disables this, but that is generally a bad
  idea, since most environment variables are required (and the ones
  that aren't should be marked as such).
* Disables the query loggers

* Upgrade meilisearch-sdk to 0.4.0 for MeiliSearch 0.16 support

* Fix swap of Forge and Fabric labeling
2020-11-05 08:38:03 -07:00
Aeledfyr d477874535 Optimize and fix some bugs in indexing (#98)
* Improve curseforge and local indexing

This should make curseforge indexing more efficient, and reuses
some of the normal local indexing for the queued indexing of
recently created mods.

* Unify impls for single and multiple routes for mods and versions

This uses the same backend for the single and multiple query
routes so that they no longer return inconsistent information.

* Cache valid curseforge mod ids to reduce request load

This caches the ids of minecraft mods and reuses them on indexing
to reduce the amount of unused addons that are returned.
2020-11-03 17:55:50 -07:00
Geometrically 5c8ffe961e Fix local indexing (#78) 2020-10-16 11:21:07 -07:00
Aeledfyr 7983e82b60 Fix some issues with search and mod creation (#77) 2020-10-16 10:04:38 -07:00
Geometrically ad29f2477e Lots of little fixes (#73)
* Lots of little fixes

* Change + Add TODOs back that were incomplete

* Fix migrations

* Run prepare

* Minor fixes

* Fix formatting

* SQLX Prepare
2020-10-14 13:19:38 -07:00
Aeledfyr 1072d1306b Make indexing date format consistent (#72) 2020-10-12 14:08:43 -07:00
Aeledfyr 781f0c843e Implement more database methods and basic API routes (#50)
* feat: Implement more database methods & add mod and version routes

* feat: Implement deleting mods/versions & implement categories

* feat: Implement routes for categories, game versions & loaders

* feat: Reorganize API routes in a (hopefully) usable way
2020-08-12 12:54:03 -07:00
Aeledfyr 8e798dde48 feat(search): Faceted search based on mod host (curse/modrinth) (#48)
This also adds a commandline argument library (gumdrop) for dealing
with indices - reseting, reconfiguring, and skipping them. I don't
know which library is best for this case, but gumdrop has shorter
compile times and many fewer dependencies than clap, which is why
I chose it.
2020-07-31 18:18:23 -07:00
Aeledfyr ff28ea8fa8 Refactor Meilisearch, update to latest SDK, and implement faceted search (#44)
* feat(indexing): Reindex curseforge & local database at an interval

* fix(indexing): Use strings for meilisearch primary key

Fixes #17 by prefixing curseforge ids with "curse-" and local ids
with "local-".

* feat(indexing): Add newly created mods to the index more quickly

* feat(indexing): Implement faceted search, update to meilisearch master

Fixes #9, but only uses faceted search for categories.  It should
be reasonably simple to add support for versions, but it may not
be as useful due to the large number of versions and the large
number of supported versions for each mod.

* feat(indexing): Allow skipping initial indexing

Co-authored-by: Geometrically <18202329+Geometrically@users.noreply.github.com>
2020-07-27 16:54:10 -07:00
AppleTheGolden ee69653a83 Switch to Postgres (#39)
* WIP Switch to Postgres

* feat(postgres): more work on porting to postgres, now compiles

* feat(docker-compose): Changed the docker-compose.yml file to use postgres.

* Update docker, documentation, gh actions...

* Remove bson dependency

* Remove bson import

* feat: move mock filehost to trait rather than cargo feature

* feat(postgres): transactions for mod creation, multipart refactor

* fix: Add Cargo.lock so that sqlx functions

* Update sqlx offline build data

* fix: Use SQLX_OFFLINE to force sqlx into offline mode for CI

* Default release channels

* feat(postgres): refactor database models to fit postgres models

* fix: Fix sqlx prepare, fix double allocation in indexing

* Add dockerfile (#40)

Co-authored-by: Charalampos Fanoulis <charalampos.fanoulis@gmail.com>

Co-authored-by: Aeledfyr <aeledfyr@gmail.com>
Co-authored-by: redblueflame <contact@redblueflame.com>
Co-authored-by: Jai A <jai.a@tuta.io>
Co-authored-by: Valentin Ricard <redblueflame1@gmail.Com>
Co-authored-by: Charalampos Fanoulis <charalampos.fanoulis@gmail.com>
2020-07-23 22:46:33 +02:00
Geometrically 39b1435725 Mod Creation (#34)
* Inital creation stuff

* File Reader

* Upload bodies

* Major rework:

* Finish Multiple Files

* Proper Error Handling

* Switch to database models

* Run formatter

* Make dependencies dependent on Versions over mods

* Fixes

* Fix clippy

* Run lint one last time

* Update src/models/mods.rs

Co-authored-by: AppleTheGolden <scotsbox@protonmail.com>

Co-authored-by: AppleTheGolden <scotsbox@protonmail.com>
2020-07-16 10:16:35 -07:00