Commit Graph

49 Commits

Author SHA1 Message Date
Geometrically
7f791d4919 Move validators to seperate thread, other fixes (#253)
* Move validators to seperate thread, other fixes

* Update rust version in Dockerfile

* Fix notifs not working

* Fix pack validator not enforcing files
2021-11-13 15:46:08 -07:00
Leo Chen
13187de97d Rustic cleanups, dedups and making the code less hard to read in general (#251)
* typos :help_me:

* (part 1/?) massive cleanup to make the code more Rust-ic and cut down heap allocations.

* (part 2/?) massive cleanup to make the code more Rust-ic and cut down heap allocations.

* (part 3/?) cut down some pretty major heap allocations here - more Bytes and BytesMuts, less Vec<u8>s

also I don't really understand why you need to `to_vec` when you don't really use it again afterwards

* (part 4/?) deduplicate error handling in backblaze logic

* (part 5/?) fixes, cleanups, refactors, and reformatting

* (part 6/?) cleanups and refactors

* remove loads of `as_str` in types that already are `Display`

* Revert "remove loads of `as_str` in types that already are `Display`"

This reverts commit 4f974310cfb167ceba03001d81388db4f0fbb509.

* reformat and move routes util to the util module

* use streams

* Run prepare + formatting issues

Co-authored-by: Jai A <jaiagr+gpg@pm.me>
Co-authored-by: Geometrically <18202329+Geometrically@users.noreply.github.com>
2021-10-11 20:26:59 -07:00
Geometrically
c87e72e08e Switch to alternate query strategy which simplifies code (#244) 2021-09-01 06:04:38 -07:00
Geometrically
efb82847cb Switch to ARRAY_AGG for database aggregations to improve peformance + fix gallery images not showing up (#242) 2021-08-31 15:29:51 -07:00
Geometrically
430c22e06e Add gallery parameter to meilisearch attributes (#227) 2021-07-27 18:10:40 -07:00
Geometrically
76b62eda3a Allow gallery featuring, add gallery images to search, rename rejection reasons, transfer ownership route (#226) 2021-07-27 16:50:07 -07:00
Geometrically
2a4caa856e More APIv2 Fixes (#210)
* Refactor search to not spam the database with queries, new utility routes for V2

* Run prepare

* More V2 Fixes

* Run prepare + formatter
2021-06-05 20:59:21 -07:00
Geometrically
157962e42a Improve peformance of search indexing, v2 fixes + new routes (#205)
* Refactor search to not spam the database with queries, new utility routes for V2

* Run prepare
2021-06-02 18:33:11 -07:00
Geometrically
16db28060c Project Types, Code Cleanup, and Rename Mods -> Projects (#192)
* Initial work for modpacks and project types

* Code cleanup, fix some issues

* Username route getting, remove pointless tests

* Base validator types + fixes

* Fix strange IML generation

* Multiple hash requests for version files

* Fix docker build (hopefully)

* Legacy routes

* Finish validator architecture

* Update rust version in dockerfile

* Added caching and fixed typo (#203)

* Added caching and fixed typo

* Fixed clippy error

* Removed log for cache

* Add final validators, fix how loaders are handled and add icons to tags

* Fix search module

* Fix parts of legacy API not working

Co-authored-by: Redblueflame <contact@redblueflame.com>
2021-05-30 15:02:07 -07:00
Mysterious_Dev
712424c339 Add alphabetically sorting (#190) 2021-04-24 19:44:11 -07:00
Jai A
e596a8f731 Fix search 2021-03-07 18:44:24 -07:00
Jai A
5b0cc73792 Change server side and client side fields to optionals 2021-03-07 17:52:04 -07:00
Jai A
2163d4465f Fix license field not working: 2021-03-07 17:07:32 -07:00
Jai A
8becf45714 Fix broken search 2021-03-07 16:39:47 -07:00
Geometrically
0ccb6cb873 Follows (#172)
* Follows initial

* Fix #171, Fix #170, Fix #169, Fix #164

* More work on follows

* Fix compile error

* Upgrade meili version, add follows to search
2021-03-04 20:35:23 -07:00
Geometrically
109d7d87bd Reports (#165)
* Reports WIP

* Finish reports

* Clippy fixes

Co-authored-by: Geometrically <geometrically@pop-os.localdomain>
2021-02-13 12:11:13 -07:00
Geometrically
174dbb5e74 Fix auth params, Add params to indexing, Order version game versions, Remove version moderation, Return donation platforms in get routes 2021-01-18 10:10:45 -07:00
Geometrically
f0b73fd696 Change index interval, add slug to search documents (#110)
* Change index interval, add slug to search documents

* Allow the removal of '@' for slug get

* Fix

* Remove name and rename side type

* Run prepare
2020-12-13 18:10:58 -07:00
Geometrically
df5684a9f8 Fix access controls (#109)
* Fix access controls

* Remove CF indexing, fix some stuff
2020-12-02 10:24:20 -07:00
Geometrically
0500994def Moderation + Mod Editing (#101)
* Moderation + Mod Editing WIP

* Run prepare, fix perms

* Make it compile

* Finish moderation and edit routes

* More fixes

* Use better queries

* Final Fixes
2020-11-15 19:58:11 -07:00
Aeledfyr
da911bfeb8 Minor fixes to orderings and permission serialization (#102)
* Fix latest_version in search results

* Handle users with invalid permissions instead of skipping them

* Specify order of some queries, fix serialization of permissions

* Run sqlx prepare
2020-11-10 09:27:36 -07:00
Aeledfyr
c8e58a1e5b Fix indexing, upgrade MeiliSearch sdk (#100)
* Hotfixes for indexing

* Handles missing INDEX_CACHE_PATH environment variable
* Exits on startup if environment variables are missing. The flag
  --allow-missing-vars disables this, but that is generally a bad
  idea, since most environment variables are required (and the ones
  that aren't should be marked as such).
* Disables the query loggers

* Upgrade meilisearch-sdk to 0.4.0 for MeiliSearch 0.16 support

* Fix swap of Forge and Fabric labeling
2020-11-05 08:38:03 -07:00
Aeledfyr
d477874535 Optimize and fix some bugs in indexing (#98)
* Improve curseforge and local indexing

This should make curseforge indexing more efficient, and reuses
some of the normal local indexing for the queued indexing of
recently created mods.

* Unify impls for single and multiple routes for mods and versions

This uses the same backend for the single and multiple query
routes so that they no longer return inconsistent information.

* Cache valid curseforge mod ids to reduce request load

This caches the ids of minecraft mods and reuses them on indexing
to reduce the amount of unused addons that are returned.
2020-11-03 17:55:50 -07:00
Geometrically
680d6c20ca Search hotfix (#86) 2020-10-19 19:46:33 -07:00
Geometrically
c886e7949e Support for using a master key (#83)
* Support for using a master key

* Expand scope of PR, add wrapper struct, add files to intitial versions/mods

* Change changelog path, run formatter

* Split file changes into different PR

* Formatting, rename main variable

Co-authored-by: Aeledfyr <aeledfyr@gmail.com>
2020-10-19 14:23:05 -07:00
Geometrically
5c8ffe961e Fix local indexing (#78) 2020-10-16 11:21:07 -07:00
Aeledfyr
7983e82b60 Fix some issues with search and mod creation (#77) 2020-10-16 10:04:38 -07:00
Geometrically
77d35b61a9 Fix indexing (#76) 2020-10-15 09:56:41 -07:00
Geometrically
ad29f2477e Lots of little fixes (#73)
* Lots of little fixes

* Change + Add TODOs back that were incomplete

* Fix migrations

* Run prepare

* Minor fixes

* Fix formatting

* SQLX Prepare
2020-10-14 13:19:38 -07:00
Aeledfyr
1072d1306b Make indexing date format consistent (#72) 2020-10-12 14:08:43 -07:00
Geometrically
b8eda40937 Fix date format (#71) 2020-10-11 11:35:50 -07:00
Geometrically
2719ae5df2 Add API routes to request multiple of an item (#70)
* Change header name

* Add default bio value

* Remove default

* Make name null

* Run prepare

* Add new API Routes for requesting multiple of an item

* Run formatter

* Simplify get mods query

* Run prepare

* Refactor to use one query for most routes, change version create route to have mod_id in data

* More fixes
2020-10-05 14:25:32 -07:00
Aeledfyr
c4fb7b7928 General cleanup: fix some bugs, some refactoring (#65)
* Merged mod file upload in version creation, mod creation and
  version file add to one function;  This makes sure that they are
  consistent
* Made some fields on `User` optional: `github_id`, `avatar_url`, `bio`.
    * We may not want to publicly show the `github_id` to everyone
      with access to the API
    * If we allow non-github users, some of those fields would be
      invalid; some oauth providers may not have avatars or bios
* Made CORS origins should configurable
* Made `--reconfigure-indices` and `--reset-indices` exit after
  completion instead of starting the server
2020-09-30 22:07:52 -07:00
Jai A
05235f8385 Implement users in API 2020-09-28 10:48:15 -07:00
Jai A
34075738ea Basic GitHub integration 2020-09-26 22:49:16 -07:00
Geometrically
88c0b8a8f0 Search fixes (#62) 2020-09-12 18:16:05 -07:00
Geometrically
e8bbc117e1 Allow for API user to change the amount of mods responded with in search (#61)
* Add more info to search route:

* Run formatter

* Allow for API user to change the amount of mods responded with in search

* Refactor SearchResults

* Fix searchresults usage
2020-09-07 11:44:21 -07:00
Geometrically
b99f45874f Add more info to search route (#60)
* Add more info to search route:

* Run formatter
2020-09-06 08:19:53 -07:00
Geometrically
2b1ed49e9a Update Meilisearch SDK (#53)
* Update Meilisearch SDK

* Run Formatter

* Fixes
2020-08-27 17:53:40 -07:00
Aeledfyr
781f0c843e Implement more database methods and basic API routes (#50)
* feat: Implement more database methods & add mod and version routes

* feat: Implement deleting mods/versions & implement categories

* feat: Implement routes for categories, game versions & loaders

* feat: Reorganize API routes in a (hopefully) usable way
2020-08-12 12:54:03 -07:00
Geometrically
e2bf474332 Version Creation (#47)
* Creation Stuff

* Make it work

* Response structs + Mod ID validation

* Run code formatter

* Push local changes

* Finish up version creation - fix comments, impl file creation

* fix: Add sqlx prepare data

Co-authored-by: Aeledfyr <aeledfyr@gmail.com>
2020-08-12 11:05:49 -07:00
Aeledfyr
8e798dde48 feat(search): Faceted search based on mod host (curse/modrinth) (#48)
This also adds a commandline argument library (gumdrop) for dealing
with indices - reseting, reconfiguring, and skipping them. I don't
know which library is best for this case, but gumdrop has shorter
compile times and many fewer dependencies than clap, which is why
I chose it.
2020-07-31 18:18:23 -07:00
Aeledfyr
ff28ea8fa8 Refactor Meilisearch, update to latest SDK, and implement faceted search (#44)
* feat(indexing): Reindex curseforge & local database at an interval

* fix(indexing): Use strings for meilisearch primary key

Fixes #17 by prefixing curseforge ids with "curse-" and local ids
with "local-".

* feat(indexing): Add newly created mods to the index more quickly

* feat(indexing): Implement faceted search, update to meilisearch master

Fixes #9, but only uses faceted search for categories.  It should
be reasonably simple to add support for versions, but it may not
be as useful due to the large number of versions and the large
number of supported versions for each mod.

* feat(indexing): Allow skipping initial indexing

Co-authored-by: Geometrically <18202329+Geometrically@users.noreply.github.com>
2020-07-27 16:54:10 -07:00
AppleTheGolden
ee69653a83 Switch to Postgres (#39)
* WIP Switch to Postgres

* feat(postgres): more work on porting to postgres, now compiles

* feat(docker-compose): Changed the docker-compose.yml file to use postgres.

* Update docker, documentation, gh actions...

* Remove bson dependency

* Remove bson import

* feat: move mock filehost to trait rather than cargo feature

* feat(postgres): transactions for mod creation, multipart refactor

* fix: Add Cargo.lock so that sqlx functions

* Update sqlx offline build data

* fix: Use SQLX_OFFLINE to force sqlx into offline mode for CI

* Default release channels

* feat(postgres): refactor database models to fit postgres models

* fix: Fix sqlx prepare, fix double allocation in indexing

* Add dockerfile (#40)

Co-authored-by: Charalampos Fanoulis <charalampos.fanoulis@gmail.com>

Co-authored-by: Aeledfyr <aeledfyr@gmail.com>
Co-authored-by: redblueflame <contact@redblueflame.com>
Co-authored-by: Jai A <jai.a@tuta.io>
Co-authored-by: Valentin Ricard <redblueflame1@gmail.Com>
Co-authored-by: Charalampos Fanoulis <charalampos.fanoulis@gmail.com>
2020-07-23 22:46:33 +02:00
Geometrically
39b1435725 Mod Creation (#34)
* Inital creation stuff

* File Reader

* Upload bodies

* Major rework:

* Finish Multiple Files

* Proper Error Handling

* Switch to database models

* Run formatter

* Make dependencies dependent on Versions over mods

* Fixes

* Fix clippy

* Run lint one last time

* Update src/models/mods.rs

Co-authored-by: AppleTheGolden <scotsbox@protonmail.com>

Co-authored-by: AppleTheGolden <scotsbox@protonmail.com>
2020-07-16 10:16:35 -07:00
Aeledfyr
b1d3e258bd fix(indexing): chunk adding documents to indexing server (#36)
This should prevent adding too many mods and going over
meilisearch's request size limit by attempting to add all mods in
one request.
2020-07-15 15:01:49 -07:00
Aeledfyr
6ff7fa74e2 Improve error handling (#33)
* refactor: improve error handling

* fix: specify bind address instead of port

* fix: remove temporary testing file

* fix(errors): change error names to snake_case

* refactor(errors): split indexing error types, remove unused errors

* feat: add env variable checking at program start

This just checks whether the enviroment variables exist and can
parse to the given type and gives a warning if they can't. This
should prevent cases where the program fails at runtime due to
checking an environment variable that doesn't exist.
2020-07-03 10:44:39 -07:00
Geometrically
91305262f1 Add Backblaze Driver (#32)
* Backblaze Driver

* Update action to work with new tests

* Fix minor issues

* Run Formatter + Switch to reqwest json parser
2020-07-02 14:00:04 +02:00
Valentin Ricard
7a6ecd86c6 Rewrite the app (#23)
* chore: Removed everything not needed, and added base for rewrite
feat(error_handling): Added 404 general cache
feat(index): Added informations about the app in the / route.

* feat(indexing): Brought back the indexing, with conditions to make it easier

* fix: Fixed build error with a forgotten call

* feat: Add Docker development enviroment (#19)

* ci: add a *lot* of new actions

* fix: rename linting action

* fix: invalid yaml begone(?)

* ci: Added cache to speed up build times

* fix(ci): 🦀ed the yaml errors

* fix(ci): fixed a missing hyphen

* ci: Added matrix of rust versions, and changed way to install rust toolchain

* fix(ci): Added names to build with the matrix so it's easier to find the source of the problem

* style(ci): Added eof lines

* refactor: Finished moving the search.rs file to a separate module.

* Search Endpoint

* refactor: Moved around functions and struct for a better understanding of what it does.

* chore: Change env default settings to resolve conversation

* refactor: Removed #[use_macros]
fix: Fixed meilisearch address from env

* chore: Added email to Aeledfyr

* fix: Brought back the dotenv variables

* style: Ran `cargo fmt`

Co-authored-by: Charalampos Fanoulis <charalampos.fanoulis@gmail.com>
Co-authored-by: Jai A <jai.a@tuta.io>
2020-06-30 19:23:52 +02:00