📦 dependencies#

Downloading the libraries a project depends on, from Smithed, Modrinth or the official ones.

download_manager#

Dynamic dependency download manager for StewBeet.

Downloads official libraries on first use via beet’s content-addressed cache (ctx.cache["stewbeet"]). Three providers are supported:

  1. Smithed API: bs.* and smithed.* libs

  2. Modrinth API: itemio (and future Modrinth libs)

  3. Static URLs: common_signals, furnace_nbt_recipes, etc.

BUILD_CACHE: dict[str, list[DownloadedLib]] = {}[source]#

Downloaded libraries of the current build, keyed by cache directory.

class DownloadedLib(
lib_ns: 'str',
name: 'str',
version: 'tuple[int, ...]',
datapack_path: 'str | None',
resource_pack_path: 'str | None',
)[source]#

Bases: object

lib_ns: str[source]#
name: str[source]#
version: tuple[int, ...][source]#
datapack_path: str | None[source]#
resource_pack_path: str | None[source]#
lookup_static(
lib_data: JsonDict,
mc_tup: tuple[int, ...],
) → tuple[tuple[int, ...], str] | None[source]#

Return (dep_ver, url) for the highest MC key <= mc_tup, or None.

mc_floats(mc_versions: list[str]) → list[float][source]#

Convert MC versions to comparable floats with stp.version_to_float, skipping infinite and unparsable ones.

Snapshots of the next version compare as newer than the current release, and release candidates as older:

>>> [round(x, 6) for x in mc_floats(["1.21.11", "26.3-snapshot-1", "26.2-rc1", "infinite"])]
[1.021011, 26.003001, 26.001999]
mc_compatible(
versions: list[JsonDict],
mc_tup: tuple[int, ...],
) → list[JsonDict][source]#

Return Smithed versions whose supports list includes mc_tup (exact), falling back to max(supports) <= mc_tup.

>>> vs = [{"name": "1.1.0", "supports": ["26.3-snapshot-1"]}, {"name": "1.0.0", "supports": ["1.21.11"]}]
>>> [v["name"] for v in mc_compatible(vs, (26, 2))]
['1.0.0']
modrinth_older_versions(
versions: list[JsonDict],
mc_ver: str,
) → list[JsonDict][source]#

Return Modrinth versions (order kept) whose game_versions are all <= mc_ver.

>>> vs = [
...     {"version_number": "1.11.0", "game_versions": ["26.3"]},
...     {"version_number": "1.10.1", "game_versions": ["1.21.8", "1.21.11"]},
... ]
>>> [v["version_number"] for v in modrinth_older_versions(vs, "26.2")]
['1.10.1']
latest_smithed_compatible(
smithed_id: str,
versions: list[JsonDict],
mc_tup: tuple[int, ...],
) → JsonDict | None[source]#

Return the latest Smithed version compatible with mc_tup, or the latest overall (with a warning) if none is.

pick_smithed_version(
smithed_id: str,
versions: list[JsonDict],
target: str | None,
mc_tup: tuple[int, ...],
) → JsonDict | None[source]#

The pinned version when there is one and it exists, else the latest one compatible with the project’s Minecraft version.

modrinth_versions(
ctx: Context,
slug: str,
mc_ver: str,
) → list[JsonDict] | None[source]#

The Modrinth versions tagged for this Minecraft version, else those made for an older one, never a newer one.

Returns:

None when the API could not be read.

get_lib_paths(
ctx: Context,
) → list[DownloadedLib][source]#

Download all is_used official libs and return their cached paths.

Results are memoised per build so weld.py and copy_to_destination can both call this without redundant work.

lib_tasks(
ctx: Context,
) → list[tuple[str, JsonDict, str]][source]#

Each library to download with its source: the official ones in use, then the load_dependencies entries naming a source.

network#

HTTP layer of the dependency downloader: cached fetches and failure diagnosis.

A download can fail because the machine is offline, because a firewall/antivirus filters HTTPS, or because the API itself is down or blocking us. These look identical from a single exception, so every failure is translated into a sentence naming the likely culprit, backed by one probe of neutral hosts.

Two of those failures clear on their own: a throttled API, and a cache file still held by another process. Both are retried with a growing delay before the build is told anything went wrong.

HEADERS: dict[str, str] = {'User-Agent': 'StewBeet'}[source]#

Headers sent with every request, some APIs reject the default urllib user-agent.

DOWNLOAD_ATTEMPTS: int = 4[source]#

Attempts allowed per download before the failure is reported.

DOWNLOAD_DELAY: float = 2.0[source]#

Seconds before the second attempt, doubling from there, so a throttled API gets 2s, 4s and 8s to calm down.

TRANSIENT_STATUS: frozenset[int] = frozenset({408, 425, 429, 500, 502, 503, 504})[source]#

HTTP statuses worth another attempt: throttling, and the server side failures that clear on their own.

PROBE_URLS: tuple[str, ...] = ('https://one.one.one.one', 'https://api.github.com')[source]#

Neutral hosts probed once when a download fails, to tell a local block apart from a failing service.

PROBE_TIMEOUT: float = 5.0[source]#

Seconds allowed to each connectivity probe before considering the host unreachable.

CONNECTIVITY: dict[str, bool] = {}[source]#

Cached probe result under the “reachable” key, filled the first time a download fails.

CONNECTIVITY_LOCK: lock = <unlocked _thread.lock object>[source]#

Guards CONNECTIVITY since libraries are resolved from several threads.

DOWNLOAD_LOCKS: dict[Path, lock] = {}[source]#

One lock per cache file. Libraries are resolved in parallel and two of them can share a download URL, which on Windows means one thread deleting or writing the file another still holds open.

DOWNLOAD_LOCKS_GUARD: lock = <unlocked _thread.lock object>[source]#

Guards DOWNLOAD_LOCKS itself, filled from the resolver threads.

HTTP_HINTS: dict[int, str] = {400: 'malformed request', 401: 'authentication required', 403: 'access denied, the API or a firewall/CDN in between is blocking this client', 404: 'not found, the pack id or slug is probably wrong', 429: 'rate limited, too many requests were sent to this API', 451: 'blocked for legal reasons', 500: 'internal server error', 502: 'bad gateway, the API is down or restarting', 503: 'service unavailable, the API is down or overloaded', 504: 'gateway timeout, the API is too slow to answer'}[source]#

Human explanation for the HTTP status codes these APIs realistically return.

SOCKET_HINTS: dict[int, str] = {13: 'socket access forbidden (EACCES), a local security policy is blocking the connection', 101: 'network unreachable (ENETUNREACH), no route to the internet', 104: 'connection reset by peer (ECONNRESET), something cut the connection mid-way', 110: 'connection timed out (ETIMEDOUT), packets are being dropped, typically by a firewall', 111: 'connection refused (ECONNREFUSED), nothing is listening on the other side', 113: 'host unreachable (EHOSTUNREACH), no route to that host', 10013: 'socket access forbidden (WinError 10013), a firewall or antivirus is blocking the connection', 10054: 'connection reset by peer (WinError 10054), something cut the connection mid-way', 10060: 'connection timed out (WinError 10060), packets are being dropped, typically by a firewall', 10061: 'connection refused (WinError 10061), nothing is listening on the other side', 10065: 'host unreachable (WinError 10065), no route to that host', 11001: 'host not found (WinError 11001), DNS could not resolve the name'}[source]#

Human explanation per socket error, keyed by Windows error code when available, else by errno.

TLS_HINTS: dict[int, str] = {9: 'the certificate is not valid yet, the system clock is probably wrong', 10: 'a certificate of the chain has expired: either the system clock is wrong, or the OS certificate store holds a stale certificate that Python picks over the valid one (on Windows, delete the expired entries from certmgr.msc > Intermediate Certification Authorities, then open the URL once in Edge)', 18: 'the server sent a self-signed certificate, HTTPS is being intercepted by an antivirus/proxy', 19: 'a self-signed certificate sits in the chain, HTTPS is being intercepted by an antivirus/proxy whose root is not trusted', 20: 'the root certificate is missing from the OS store, or HTTPS is being intercepted by an antivirus/proxy', 62: 'the certificate does not match this host name, a proxy or captive portal is answering instead'}[source]#

Human explanation per OpenSSL verification error, keyed by X509_V_ERR code.

TLS_WORKAROUND: str = "setting the SSL_CERT_FILE environment variable to the path printed by 'python -m certifi' bypasses the OS store"[source]#

Appended to certificate failures, since trusting certifi’s bundle instead of the OS store fixes most of them.

exception TransientDownloadError[source]#

Bases: Exception

A failure worth another attempt, wrapping the real one as its cause.

The retry decorator selects on the exception type, while the diagnosis needs the original exception to name the culprit, so the two are kept apart rather than reported as one another.

is_transient(exc: BaseException) → bool[source]#

Whether a download exception is worth another attempt: a throttled API, a cut connection, or a locked cache file.

>>> is_transient(HTTPError("", 429, "Too Many Requests", {}, None))
True
>>> is_transient(HTTPError("", 404, "Not Found", {}, None))
False
>>> is_transient(PermissionError(13, "being used by another process"))
True
>>> is_transient(URLError(ConnectionResetError()))
True
download_lock(target: Path) → lock[source]#

The lock owning target, created on first use.

internet_reachable() → bool[source]#

Return whether any neutral host answers, probing at most once per build.

host_answers(url: str) → bool[source]#

Return whether url answered anything at all, an HTTP error still counting as an answer.

connectivity_verdict() → str[source]#

Suffix naming the culprit, since a dead service and a blocked machine look identical from one failure.

socket_hint(exc: OSError) → str[source]#

Explain a socket-level failure, keyed by Windows error code when there is one, else by errno.

describe_network_error(url: str, exc: BaseException) → str[source]#

Turn a raw download exception into a one-line diagnosis naming the likely culprit.

Parameters:
  • url – The URL that failed, used to report the host name.

  • exc – The exception raised while downloading.

Returns:

The diagnosis, ex: “HTTP 503 Service Unavailable from api.smithed.dev: the API is down or overloaded”

Return type:

str

>>> describe_network_error(
...     "https://api.smithed.dev/v2/packs/x", HTTPError("", 404, "Not Found", {}, None)
... )
'HTTP 404 Not Found from api.smithed.dev: not found, ...'
describe_body(text: str) → str[source]#

Describe a response that is not the expected JSON, so a proxy or captive-portal page is obvious.

>>> describe_body("<html>Blocked</html>")
'HTML received instead of JSON (proxy, captive portal or error page): <html>Blocked</html>'
attempt_download(
cache: Cache,
url: str,
target: Path,
) → Path[source]#

One download attempt, never letting a previously failed one masquerade as a hit.

beet creates the destination file before fetching, so a failed download leaves a 0-byte file that every later build happily reuses, hiding the real error behind a bogus parsing failure.

download_to_cache(
ctx: Context,
url: str,
path: Path | None = None,
) → Path[source]#

Download url through beet’s cache, one thread at a time per file, retrying what can still succeed.

cached_json(
ctx: Context,
url: str,
) → Any | None[source]#

Download and parse a JSON document, reporting why it failed instead of a bare parsing error.

cached_zip(
ctx: Context,
url: str,
) → Path | None[source]#

Download url via beet cache, ensuring the file has a .zip suffix.

official_libs#

detection_markers(project_id: str) → dict[str, str][source]#

Map every text marker to the library it proves the use of, skipping the pack being built.

Parameters:

project_id – Project id of the pack being built, so a library never detects itself.

Returns:

Marker to look for in function sources, mapped to its library namespace.

Return type:

dict[str, str]

>>> detection_markers("my_pack")["#bs.math:"]
'bs.math'
>>> "#bs.math:" in detection_markers("bookshelf")
False