Skip to content

Study abroad schema #211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
NathanP9000 wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Study abroad schema #211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a123fd3
Scraping program discovery page. Next will go to a returned program a…
NathanP9000 Apr 8, 2026
a6d2bb8
Fetch Brochure function works
NathanP9000 Apr 9, 2026
20deabc
Switch to nebula-api rest/schema module path
NathanP9000 Oct 21, 2026
2197099
Scrape and parse study abroad programs
NathanP9000 Oct 21, 2026
3dbf498
Parse brochures into updated study abroad schema
NathanP9000 Oct 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
# Scrapers
# Optional: path to a Chrome/Chromium binary for ChromeDP.
# Leave blank to auto-detect google-chrome, chromium, or chromium-browser on PATH.
# Set this if auto-detection picks a browser that fails to launch (e.g. snap-installed Chromium).
CHROME_PATH=
LOGIN_NETID=
LOGIN_PASSWORD=
LOGIN_ASTRA_USERNAME=
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ Run the tool by changing directory using `cd` to the `api-tools` directory and r
| `./api-tools -scrape -map` | Scrapes UTD Map data. |
| `./api-tools -scrape -mazevo` | Scrapes Mazevo data. |
| `./api-tools -scrape -profiles` | Scrapes UTD professor profiles. |
| `./api-tools -scrape -studyAbroad` | Scrapes study abroad program brochures and discovery records to `[outDir]/studyAbroad`.<br>• Set `PROGRAM_IDS` to a comma-separated list to scrape specific programs instead of all of them. |
| `./api-tools -scrape -headless` | Runs ChromeDP in headless mode. |
| `./api-tools -o [directory]` | Sets output directory (default: `./data`). |

Expand All @@ -87,6 +88,7 @@ Run the tool by changing directory using `cd` to the `api-tools` directory and r
| `./api-tools -parse -degrees` | Parses degrees from HTML. |
| `./api-tools -parse -map` | Parses UTD Map data. |
| `./api-tools -parse -mazevo` | Parses Mazevo data. |
| `./api-tools -parse -studyAbroad` | Combines scraped study abroad brochures and discovery records into `studyAbroad.json`. |
| `./api-tools -parse -skipv` | Skips post-parse validation (**use with caution**). |


Expand Down
76 changes: 38 additions & 38 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,81 +1,81 @@
module github.com/UTDNebula/api-tools

go 1.26
go 1.26.0

require (
github.com/PuerkitoBio/goquery v1.12.0
github.com/UTDNebula/nebula-api/api v0.0.0-20260525053158-3209b0868dcf
github.com/UTDNebula/nebula-api v1.2.1-0.20261022120845-36f2148b044a
github.com/chromedp/cdproto v0.0.0-20260321001828-e3e3800016bc
github.com/chromedp/chromedp v0.15.1
github.com/dongri/phonenumber v0.1.12
github.com/getsentry/sentry-go v0.46.2
github.com/getsentry/sentry-go v0.48.0
github.com/google/go-cmp v0.7.0
github.com/joho/godotenv v1.5.1
github.com/valyala/fastjson v1.6.10
go.mongodb.org/mongo-driver v1.17.9
golang.org/x/net v0.55.0
golang.org/x/text v0.39.0
golang.org/x/net v0.58.0
golang.org/x/text v0.41.0
google.golang.org/genai v1.51.0
)

require (
cel.dev/expr v0.25.1 // indirect
cel.dev/expr v0.25.3 // indirect
cloud.google.com/go v0.123.0 // indirect
cloud.google.com/go/auth v0.19.0 // indirect
cloud.google.com/go/auth v0.23.2 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
cloud.google.com/go/iam v1.6.0 // indirect
cloud.google.com/go/monitoring v1.24.3 // indirect
cloud.google.com/go/storage v1.61.3 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.55.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.55.0 // indirect
cloud.google.com/go/iam v1.13.0 // indirect
cloud.google.com/go/monitoring v1.30.0 // indirect
cloud.google.com/go/storage v1.65.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.36.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.60.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.60.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chromedp/sysutil v1.1.0 // indirect
github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.37.0 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.39.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.3.3 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
github.com/go-json-experiment/json v0.0.0-20260214004413-d219187c3433 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/gobwas/httphead v0.1.0 // indirect
github.com/gobwas/pool v0.2.1 // indirect
github.com/gobwas/ws v1.4.0 // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.14 // indirect
github.com/googleapis/gax-go/v2 v2.20.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.21 // indirect
github.com/googleapis/gax-go/v2 v2.24.0 // indirect
github.com/gorilla/websocket v1.5.3 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/montanaflynn/stats v0.9.0 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/montanaflynn/stats v0.12.4 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/spiffe/go-spiffe/v2 v2.6.0 // indirect
github.com/spiffe/go-spiffe/v2 v2.8.1 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.2.0 // indirect
github.com/xdg-go/stringprep v1.0.4 // indirect
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.42.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.67.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
golang.org/x/crypto v0.51.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.45.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.70.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 // indirect
go.opentelemetry.io/otel v1.45.0 // indirect
go.opentelemetry.io/otel/metric v1.45.0 // indirect
go.opentelemetry.io/otel/sdk v1.45.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.45.0 // indirect
go.opentelemetry.io/otel/trace v1.45.0 // indirect
golang.org/x/crypto v0.55.0 // indirect
golang.org/x/oauth2 v0.36.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/api v0.273.0 // indirect
google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260319201613-d00831a3d3e7 // indirect
google.golang.org/grpc v1.79.3 // indirect
google.golang.org/protobuf v1.36.11 // indirect
google.golang.org/api v0.293.0 // indirect
google.golang.org/genproto v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
google.golang.org/grpc v1.83.2 // indirect
google.golang.org/protobuf v1.36.12 // indirect
)
Loading
Loading