You've already forked AstralRinth
forked from didirus/AstralRinth
Technical review queue (#4775)
* chore: fix typo in status message * feat(labrinth): overhaul malware scanner report storage and routes * chore: address some review comments * feat: add Delphi to Docker Compose `with-delphi` profile * chore: fix unused import Clippy lint * feat(labrinth/delphi): use PAT token authorization with project read scopes * chore: expose file IDs in version queries * fix: accept null decompiled source payloads from Delphi * tweak(labrinth): expose base62 file IDs more consistently for Delphi * feat(labrinth/delphi): support new Delphi report severity field * chore(labrinth): run `cargo sqlx prepare` to fix Docker build errors * tweak: add route for fetching Delphi issue type schema, abstract Labrinth away from issue types * chore: run `cargo sqlx prepare` * chore: fix typo on frontend generated state file message * feat: update to use new Delphi issue schema * wip: tech review endpoints * wip: add ToSchema for dependent types * wip: report issues return * wip * wip: returning more data * wip * Fix up db query * Delphi configuration to talk to Labrinth * Get Delphi working with Labrinth * Add Delphi dummy fixture * Better Delphi logging * Improve utoipa for tech review routes * Add more sorting options for tech review queue * Oops join * New routes for fetching issues and reports * Fix which kind of ID is returned in tech review endpoints * Deduplicate tech review report rows * Reduce info sent for projects * Fetch more thread info * Address PR comments * fix ci * fix postgres version mismatch * fix version creation * Implement routes * fix up tech review * Allow adding a moderation comment to Delphi rejections * fix up rebase * exclude rejected projects from tech review * add status change msg to tech review thread * cargo sqlx prepare * also ignore withheld projects * More filtering on issue search * wip: report routes * Fix up for build * cargo sqlx prepare * fix thread message privacy * New tech review search route * submit route * details have statuses now * add default to drid status * dedup issue details * fix sqlx query on empty files * fixes * Dedupe issue detail statuses and message on entering tech rev * Fix qa issues * Fix qa issues * fix review comments * typos * fix ci * feat: tech review frontend (#4781) * chore: fix typo in status message * feat(labrinth): overhaul malware scanner report storage and routes * chore: address some review comments * feat: add Delphi to Docker Compose `with-delphi` profile * chore: fix unused import Clippy lint * feat(labrinth/delphi): use PAT token authorization with project read scopes * chore: expose file IDs in version queries * fix: accept null decompiled source payloads from Delphi * tweak(labrinth): expose base62 file IDs more consistently for Delphi * feat(labrinth/delphi): support new Delphi report severity field * chore(labrinth): run `cargo sqlx prepare` to fix Docker build errors * tweak: add route for fetching Delphi issue type schema, abstract Labrinth away from issue types * chore: run `cargo sqlx prepare` * chore: fix typo on frontend generated state file message * feat: update to use new Delphi issue schema * wip: tech review endpoints * wip: add ToSchema for dependent types * wip: report issues return * wip * wip: returning more data * wip * Fix up db query * Delphi configuration to talk to Labrinth * Get Delphi working with Labrinth * Add Delphi dummy fixture * Better Delphi logging * Improve utoipa for tech review routes * Add more sorting options for tech review queue * Oops join * New routes for fetching issues and reports * Fix which kind of ID is returned in tech review endpoints * Deduplicate tech review report rows * Reduce info sent for projects * Fetch more thread info * Address PR comments * fix ci * fix ci * fix postgres version mismatch * fix version creation * Implement routes * feat: batch scan alert * feat: layout * feat: introduce surface variables * fix: theme selector * feat: rough draft of tech review card * feat: tab switcher * feat: batch scan btn * feat: api-client module for tech review * draft: impl * feat: auto icons * fix: layout issues * feat: fixes to code blocks + flag labels * feat: temp remove mock data * fix: search sort types * fix: intl & lint * chore: re-enable mock data * fix: flag badges + auto open first issue in file tab * feat: update for new routes * fix: more qa issues * feat: lazy load sources * fix: re-enable auth middleware * feat: impl threads * fix: lint & severity * feat: download btn + switch to using NavTabs with new local mode option * feat: re-add toplevel btns * feat: reports page consistency * fix: consistency on project queue * fix: icons + sizing * fix: colors and gaps * fix: impl endpoints * feat: load all flags on file tab * feat: thread generics changes * feat: more qa * feat: fix collapse * fix: qa * feat: msg modal * fix: ISO import * feat: qa fixes * fix: empty state basic * fix: collapsible region * fix: collapse thread by default * feat: rough draft of new process/flow * fix labrinth build * fix thread message privacy * New tech review search route * feat: qa fixes * feat: QA changes * fix: verdict on detail not whole issue * fix: lint + intl * fix: lint * fix: thread message for tech rev verdict * feat: use anim frames * fix: exports + typecheck * polish: qa changes * feat: qa * feat: qa polish * feat: fix malic modal * fix: lint * fix: qa + lint * fix: pagination * fix: lint * fix: qa * intl extract * fix ci --------- Signed-off-by: Calum H. <contact@cal.engineer> Co-authored-by: Alejandro González <me@alegon.dev> Co-authored-by: aecsocket <aecsocket@tutanota.com> --------- Signed-off-by: Calum H. <contact@cal.engineer> Co-authored-by: Alejandro González <me@alegon.dev> Co-authored-by: Calum H. <contact@cal.engineer>
This commit is contained in:
aecsocket
266
apps/labrinth/src/database/models/delphi_report_item.rs
Normal file
266
apps/labrinth/src/database/models/delphi_report_item.rs
Normal file
@@ -0,0 +1,266 @@
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
fmt::{self, Display, Formatter},
|
||||
};
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::types::Json;
|
||||
|
||||
use crate::database::models::{
|
||||
DBFileId, DBProjectId, DatabaseError, DelphiReportId,
|
||||
DelphiReportIssueDetailsId, DelphiReportIssueId,
|
||||
};
|
||||
|
||||
/// A Delphi malware analysis report for a project version file.
|
||||
///
|
||||
/// Malware analysis reports usually belong to a specific project file,
|
||||
/// but they can get orphaned if the versions they belong to are deleted.
|
||||
/// Thus, deleting versions does not delete these reports.
|
||||
#[derive(Serialize)]
|
||||
pub struct DBDelphiReport {
|
||||
pub id: DelphiReportId,
|
||||
pub file_id: Option<DBFileId>,
|
||||
/// A sequential, monotonically increasing version number for the
|
||||
/// Delphi version that generated this report.
|
||||
pub delphi_version: i32,
|
||||
pub artifact_url: String,
|
||||
pub created: DateTime<Utc>,
|
||||
pub severity: DelphiSeverity,
|
||||
}
|
||||
|
||||
impl DBDelphiReport {
|
||||
pub async fn upsert(
|
||||
&self,
|
||||
transaction: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||
) -> Result<DelphiReportId, DatabaseError> {
|
||||
Ok(DelphiReportId(sqlx::query_scalar!(
|
||||
"
|
||||
INSERT INTO delphi_reports (file_id, delphi_version, artifact_url, severity)
|
||||
VALUES ($1, $2, $3, $4)
|
||||
ON CONFLICT (file_id, delphi_version) DO UPDATE SET
|
||||
delphi_version = $2, artifact_url = $3, created = CURRENT_TIMESTAMP, severity = $4
|
||||
RETURNING id
|
||||
",
|
||||
self.file_id as Option<DBFileId>,
|
||||
self.delphi_version,
|
||||
self.artifact_url,
|
||||
self.severity as DelphiSeverity,
|
||||
)
|
||||
.fetch_one(&mut **transaction)
|
||||
.await?))
|
||||
}
|
||||
}
|
||||
|
||||
/// A severity level reported by Delphi.
|
||||
#[derive(
|
||||
Deserialize,
|
||||
Serialize,
|
||||
Debug,
|
||||
Clone,
|
||||
Copy,
|
||||
PartialEq,
|
||||
Eq,
|
||||
Hash,
|
||||
sqlx::Type,
|
||||
utoipa::ToSchema,
|
||||
)]
|
||||
// The canonical serialized form of this enum is the snake_case representation.
|
||||
// We add `alias`es so we can deserialize it from how Delphi sends it,
|
||||
// which follows the Java conventions of `SCREAMING_SNAKE_CASE`.
|
||||
#[serde(rename_all = "snake_case")]
|
||||
#[sqlx(type_name = "delphi_severity", rename_all = "snake_case")]
|
||||
pub enum DelphiSeverity {
|
||||
#[serde(alias = "LOW")]
|
||||
Low,
|
||||
#[serde(alias = "MEDIUM")]
|
||||
Medium,
|
||||
#[serde(alias = "HIGH")]
|
||||
High,
|
||||
#[serde(alias = "SEVERE")]
|
||||
Severe,
|
||||
}
|
||||
|
||||
/// An issue found in a Delphi report. Every issue belongs to a report,
|
||||
/// and a report can have zero, one, or more issues attached to it.
|
||||
#[derive(Deserialize, Serialize)]
|
||||
pub struct DBDelphiReportIssue {
|
||||
pub id: DelphiReportIssueId,
|
||||
pub report_id: DelphiReportId,
|
||||
pub issue_type: String,
|
||||
}
|
||||
|
||||
/// A status a Delphi report issue can have.
|
||||
#[derive(
|
||||
Deserialize,
|
||||
Serialize,
|
||||
Debug,
|
||||
Clone,
|
||||
Copy,
|
||||
PartialEq,
|
||||
Eq,
|
||||
Hash,
|
||||
sqlx::Type,
|
||||
utoipa::ToSchema,
|
||||
)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
#[sqlx(type_name = "delphi_report_issue_status", rename_all = "snake_case")]
|
||||
pub enum DelphiStatus {
|
||||
/// The issue is pending review by the moderation team.
|
||||
Pending,
|
||||
/// The issue has been rejected (i.e., reviewed as a false positive).
|
||||
/// The affected artifact has thus been verified to be clean, other issues
|
||||
/// with it notwithstanding.
|
||||
Safe,
|
||||
/// The issue has been approved (i.e., reviewed as a valid, true positive).
|
||||
/// The affected artifact has thus been verified to be potentially malicious.
|
||||
Unsafe,
|
||||
}
|
||||
|
||||
impl Display for DelphiStatus {
|
||||
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
|
||||
self.serialize(f)
|
||||
}
|
||||
}
|
||||
|
||||
/// What verdict a moderator can give to a project flagged for technical review.
|
||||
#[derive(
|
||||
Deserialize,
|
||||
Serialize,
|
||||
Debug,
|
||||
Clone,
|
||||
Copy,
|
||||
PartialEq,
|
||||
Eq,
|
||||
Hash,
|
||||
sqlx::Type,
|
||||
utoipa::ToSchema,
|
||||
)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DelphiVerdict {
|
||||
/// The issue has been rejected (i.e., reviewed as a false positive).
|
||||
/// The affected artifact has thus been verified to be clean, other issues
|
||||
/// with it notwithstanding.
|
||||
Safe,
|
||||
/// The issue has been approved (i.e., reviewed as a valid, true positive).
|
||||
/// The affected artifact has thus been verified to be potentially malicious.
|
||||
Unsafe,
|
||||
}
|
||||
|
||||
/// An order in which Delphi report issues can be sorted during queries.
|
||||
#[derive(Deserialize, Serialize, Debug, Clone, Copy, PartialEq, Eq, Hash)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum DelphiReportListOrder {
|
||||
CreatedAsc,
|
||||
CreatedDesc,
|
||||
PendingStatusFirst,
|
||||
SeverityAsc,
|
||||
SeverityDesc,
|
||||
}
|
||||
|
||||
impl Display for DelphiReportListOrder {
|
||||
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
|
||||
self.serialize(f)
|
||||
}
|
||||
}
|
||||
|
||||
/// A result returned from a Delphi report issue query, slightly
|
||||
/// denormalized with related entity information for ease of
|
||||
/// consumption by clients.
|
||||
#[derive(Serialize)]
|
||||
pub struct DelphiReportIssueResult {
|
||||
pub issue: DBDelphiReportIssue,
|
||||
pub report: DBDelphiReport,
|
||||
pub details: Vec<ReportIssueDetail>,
|
||||
pub project_id: Option<DBProjectId>,
|
||||
pub project_published: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
impl DBDelphiReportIssue {
|
||||
pub async fn insert(
|
||||
&self,
|
||||
transaction: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||
) -> Result<DelphiReportIssueId, DatabaseError> {
|
||||
Ok(DelphiReportIssueId(
|
||||
sqlx::query_scalar!(
|
||||
"
|
||||
INSERT INTO delphi_report_issues (report_id, issue_type)
|
||||
VALUES ($1, $2)
|
||||
RETURNING id
|
||||
",
|
||||
self.report_id as DelphiReportId,
|
||||
self.issue_type,
|
||||
)
|
||||
.fetch_one(&mut **transaction)
|
||||
.await?,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
/// The details of a Delphi report issue, which contain data about a
|
||||
/// Java class affected by it. Every Delphi report issue details object
|
||||
/// belongs to a specific issue, and an issue can have zero, one, or
|
||||
/// more details attached to it. (Some issues may be artifact-wide,
|
||||
/// or otherwise not really specific to any particular class.)
|
||||
#[derive(
|
||||
Debug, Clone, Deserialize, Serialize, utoipa::ToSchema, sqlx::FromRow,
|
||||
)]
|
||||
pub struct ReportIssueDetail {
|
||||
/// ID of this issue detail.
|
||||
pub id: DelphiReportIssueDetailsId,
|
||||
/// ID of the issue this detail belongs to.
|
||||
pub issue_id: DelphiReportIssueId,
|
||||
/// Opaque identifier for where this issue detail is located, relative to
|
||||
/// the file scanned.
|
||||
///
|
||||
/// This acts as a stable identifier for an issue detail, even across
|
||||
/// different versions of the same file.
|
||||
pub key: String,
|
||||
/// Name of the Java class path in which this issue was found.
|
||||
pub file_path: String,
|
||||
/// Decompiled, pretty-printed source of the Java class.
|
||||
pub decompiled_source: Option<String>,
|
||||
/// Extra detail-specific info for this detail.
|
||||
#[sqlx(json)]
|
||||
pub data: HashMap<String, serde_json::Value>,
|
||||
/// How important is this issue, as flagged by Delphi?
|
||||
pub severity: DelphiSeverity,
|
||||
/// Has this issue detail been marked as safe or unsafe?
|
||||
pub status: DelphiStatus,
|
||||
}
|
||||
|
||||
impl ReportIssueDetail {
|
||||
pub async fn insert(
|
||||
&self,
|
||||
transaction: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||
) -> Result<DelphiReportIssueDetailsId, DatabaseError> {
|
||||
Ok(DelphiReportIssueDetailsId(sqlx::query_scalar!(
|
||||
"
|
||||
INSERT INTO delphi_report_issue_details (issue_id, key, file_path, decompiled_source, data, severity)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
RETURNING id
|
||||
",
|
||||
self.issue_id as DelphiReportIssueId,
|
||||
self.key,
|
||||
self.file_path,
|
||||
self.decompiled_source,
|
||||
sqlx::types::Json(&self.data) as Json<&HashMap<String, serde_json::Value>>,
|
||||
self.severity as DelphiSeverity,
|
||||
)
|
||||
.fetch_one(&mut **transaction)
|
||||
.await?))
|
||||
}
|
||||
|
||||
pub async fn remove_all_by_issue_id(
|
||||
issue_id: DelphiReportIssueId,
|
||||
transaction: &mut sqlx::Transaction<'_, sqlx::Postgres>,
|
||||
) -> Result<u64, DatabaseError> {
|
||||
Ok(sqlx::query!(
|
||||
"DELETE FROM delphi_report_issue_details WHERE issue_id = $1",
|
||||
issue_id as DelphiReportIssueId,
|
||||
)
|
||||
.execute(&mut **transaction)
|
||||
.await?
|
||||
.rows_affected())
|
||||
}
|
||||
}
|
||||
@@ -94,7 +94,7 @@ macro_rules! generate_bulk_ids {
|
||||
|
||||
macro_rules! impl_db_id_interface {
|
||||
($id_struct:ident, $db_id_struct:ident, $(, generator: $generator_function:ident @ $db_table:expr, $(bulk_generator: $bulk_generator_function:ident,)?)?) => {
|
||||
#[derive(Copy, Clone, Debug, Type, Serialize, Deserialize, PartialEq, Eq, Hash)]
|
||||
#[derive(Copy, Clone, Debug, Type, Serialize, Deserialize, PartialEq, Eq, Hash, utoipa::ToSchema)]
|
||||
#[sqlx(transparent)]
|
||||
pub struct $db_id_struct(pub i64);
|
||||
|
||||
@@ -140,8 +140,8 @@ macro_rules! db_id_interface {
|
||||
};
|
||||
}
|
||||
|
||||
macro_rules! short_id_type {
|
||||
($name:ident) => {
|
||||
macro_rules! id_type {
|
||||
($name:ident as $type:ty) => {
|
||||
#[derive(
|
||||
Copy,
|
||||
Clone,
|
||||
@@ -152,9 +152,10 @@ macro_rules! short_id_type {
|
||||
Eq,
|
||||
PartialEq,
|
||||
Hash,
|
||||
utoipa::ToSchema,
|
||||
)]
|
||||
#[sqlx(transparent)]
|
||||
pub struct $name(pub i32);
|
||||
pub struct $name(pub $type);
|
||||
};
|
||||
}
|
||||
|
||||
@@ -268,14 +269,17 @@ db_id_interface!(
|
||||
generator: generate_affiliate_code_id @ "affiliate_codes",
|
||||
);
|
||||
|
||||
short_id_type!(CategoryId);
|
||||
short_id_type!(GameId);
|
||||
short_id_type!(LinkPlatformId);
|
||||
short_id_type!(LoaderFieldEnumId);
|
||||
short_id_type!(LoaderFieldEnumValueId);
|
||||
short_id_type!(LoaderFieldId);
|
||||
short_id_type!(LoaderId);
|
||||
short_id_type!(NotificationActionId);
|
||||
short_id_type!(ProjectTypeId);
|
||||
short_id_type!(ReportTypeId);
|
||||
short_id_type!(StatusId);
|
||||
id_type!(CategoryId as i32);
|
||||
id_type!(GameId as i32);
|
||||
id_type!(LinkPlatformId as i32);
|
||||
id_type!(LoaderFieldEnumId as i32);
|
||||
id_type!(LoaderFieldEnumValueId as i32);
|
||||
id_type!(LoaderFieldId as i32);
|
||||
id_type!(LoaderId as i32);
|
||||
id_type!(NotificationActionId as i32);
|
||||
id_type!(ProjectTypeId as i32);
|
||||
id_type!(ReportTypeId as i32);
|
||||
id_type!(StatusId as i32);
|
||||
id_type!(DelphiReportId as i64);
|
||||
id_type!(DelphiReportIssueId as i64);
|
||||
id_type!(DelphiReportIssueDetailsId as i64);
|
||||
|
||||
@@ -4,6 +4,7 @@ pub mod affiliate_code_item;
|
||||
pub mod categories;
|
||||
pub mod charge_item;
|
||||
pub mod collection_item;
|
||||
pub mod delphi_report_item;
|
||||
pub mod flow_item;
|
||||
pub mod friend_item;
|
||||
pub mod ids;
|
||||
|
||||
@@ -11,7 +11,7 @@ pub struct ThreadBuilder {
|
||||
pub report_id: Option<DBReportId>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Serialize)]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, utoipa::ToSchema)]
|
||||
pub struct DBThread {
|
||||
pub id: DBThreadId,
|
||||
|
||||
@@ -30,7 +30,7 @@ pub struct ThreadMessageBuilder {
|
||||
pub hide_identity: bool,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone)]
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, utoipa::ToSchema)]
|
||||
pub struct DBThreadMessage {
|
||||
pub id: DBThreadMessageId,
|
||||
pub thread_id: DBThreadId,
|
||||
|
||||
@@ -6,6 +6,7 @@ use crate::database::models::loader_fields::{
|
||||
};
|
||||
use crate::database::redis::RedisPool;
|
||||
use crate::models::projects::{FileType, VersionStatus};
|
||||
use crate::routes::internal::delphi::DelphiRunParameters;
|
||||
use chrono::{DateTime, Utc};
|
||||
use dashmap::{DashMap, DashSet};
|
||||
use futures::TryStreamExt;
|
||||
@@ -164,6 +165,17 @@ impl VersionFileBuilder {
|
||||
.await?;
|
||||
}
|
||||
|
||||
if let Err(err) = crate::routes::internal::delphi::run(
|
||||
&mut **transaction,
|
||||
DelphiRunParameters {
|
||||
file_id: file_id.into(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Error submitting new file to Delphi: {err}");
|
||||
}
|
||||
|
||||
Ok(file_id)
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user