ADR proposals for how learning packages relate to courses/runs [FC-0138]

[bradenmacdonald]

Before I propose File Components, I want to settle some open questions around Learning Packages, and copy-on-write with these two ADRs.

View the rendered ADRs:

• 11. Do not implement copy-on-write
• 12. Course Learning Packages

AI disclosure: Claude helped me with research, writing, and formatting the RST. I wrote and edited much of the content myself.

[openedx-webhooks]

Thanks for the pull request, @bradenmacdonald!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ormsbee]

I'll leave a more detailed review in the next couple of days, but some high level thoughts/reactions:

Media de-duplication

1. I think the blob_namespace idea is a clever workaround for our biggest storage issues, and I really like it.
2. I do wonder if we should just force the namespace to a shared org level for new entries though, since there seems to be very little downside to it.
3. I'm trying to think through whether it's really okay for it to be immutable, what the use cases might be for wanting to change it, or what cleanup looks like in the shared scenario.
4. I might quibble with the naming a bit. Something like media_file_namespace might make the intent clearer.

That being said, I'm broadly supportive of this, and I think whatever wrinkles there are can be worked out.

Dependency relationship between openedx_content and openedx_catalog

The Pathways ADR assumes the opposite relationship between openedx_content and openedx_catalog. (In retrospect, we should have spun off a separate ADR for this decision in one of those applets.) I was thinking that any context -> content mappings could happen from the openedx_content side.

1:1 relationship between CourseRun and LearningPackage

This is the one that I'm most concerned about. We are aiming for a future where we can have wildly flexible arrangements of content. I don't want to assume that the authoring and learning contexts are 1:1. I would like to be in a place where someday where we can export a self-contained LearningPackage that contains a Pathway, its half dozen courses, and a bunch of smaller things. Maybe this means that we'll need to make a new model to capture namespaced identifiers to resolve potential conflicts between different course runs, but I think this sort of thing is an acceptable trade-off.

Note that I'm not saying that we should force all runs of a course into the same LearningPackage, or try to force re-runs to share the same PublishableEntities. That presents permissions headaches for interconnected content. But I do think that we should leave the door open for scenarios where there is a team working on content in a more centralized way, and it makes sense for that team to have such broad permissions across multiple courses or smaller learning contexts (e.g. bootcamps). We could even have scenarios where people are doing a library sync for the course run as a whole--if it's mapped to the same blob_namespace, that will be relatively cheap.

[ChrisChV]

A rerun creates only the container spine, pointing at the previous run's components pinned to their published versions, and forks a component into its own scope on first edit.

I feel this approach can be decoupled from the decision not to use a LearningPackage per CatalogCourse. My question is: can't this approach be used with "One LearningPackage per CourseRun"?

[bradenmacdonald]

That's true, and it does sound like we may still want something like this in the future. Let me think about it.

[ChrisChV]

No code may depend on two learning packages sharing a blob namespace; it is a storage optimization only

How will this be achieved in practice?

[bradenmacdonald]

I don't think we can strictly enforce it in practice, though we could try to write test fixtures that include two runs that don't share a namespace, to try to catch any issues. That would likely live in the platform tests though.

[ChrisChV]

Since the LearningPackage can be null, was there any consideration of deferring the course copy when creating a rerun? That is, the LearningPackage would remain null until a specific trigger occurs (opening the course in Studio comes to mind, though that is open for discussion). Only after the trigger is met would the course copy be generated and the LearningPackage set.

[bradenmacdonald]

That's an interesting idea. We certainly could do that, and it would make sense to let people start enrolling in the new version while deferring the actual creation of it so that it can be based on the very latest version of the previous course once it actually gets created. I think it would be possible, but it's not something I planned for here.

[bradenmacdonald]

The Pathways ADR assumes the opposite relationship between openedx_content and openedx_catalog. (In retrospect, we should have spun off a separate ADR for this decision in one of those applets.) I was thinking that any context -> content mappings could happen from the openedx_content side.

Why don't I spin that off to a separate ADR then, and we can figure it out in a dedicated PR.

It is a bit of a problem if Pathways are already doing the opposite of what libraries are doing.

Edit: here's the PR: #818

[kdmccormick]

I do wonder if we should just force the namespace to a shared org level for new entries though, since there seems to be very little downside to it.

This seems simpler and just as good.

Confirming my understanding: We would have a site-wide namespace if not for the possibility of a birthday attack, right?

[kdmccormick]

This is the one that I'm most concerned about. We are aiming for a future where we can have wildly flexible arrangements of content. I don't want to assume that the authoring and learning contexts are 1:1. I would like to be in a place where someday where we can export a self-contained LearningPackage that contains a Pathway, its half dozen courses, and a bunch of smaller things.

+1

Maybe this means that we'll need to make a new model to capture namespaced identifiers to resolve potential conflicts between different course runs, but I think this sort of thing is an acceptable trade-off.

Could this be the Usage table?

[kdmccormick]

ADR 11 looks good to me, thank you! ✅

Hadn't considered "Containers cannot be shared anyway" but that seems like a slam-dunk reason on its own.

[bradenmacdonald]

Yeah. I know it's a bit unusual to have an ADR about not doing something and keeping the status quo, but my head kept trying to loop back to some variant of that copy on write idea, so I wanted to document clearly why we should go in a different direction. And the container thing was something I hadn't realized before but is definitely a strong reason.

[bradenmacdonald]

We would have a site-wide namespace if not for the possibility of a birthday attack, right?

I wouldn't have thought of a birthday attack as the main risk; based on our usage of BLAKE2b truncated to 20 bytes, Claude says that finding any two files with the same digest costs about 2^80 hash evaluations which is "infeasible for realistic attackers today, but below the margin you would choose if you were designing for cross-tenant safety."

More trivially exploitable is an existence oracle where org B can determine if org A already uploaded a file with specific contents (specific hash) by looking for how quickly it is "uploaded" and stored, or other details of the write request. In our context, that's not exactly a worrying possibility.

Generally I think the main reason to prefer an org-specific namespace instead of site-wide is just the general principle of tenant isolation and also to a lesser extent sharding of the content (even if it's purely virtual in this case). Following that principle in general provides a more robust system that's less vulnerable to security issues, and it's always less of a concern when security is compromised within a tenant's data rather than across tenants, even if the security risks are pretty low in this case.

[bradenmacdonald]

Oh: the other big argument for maintaining per-org separation: combining all assets into a single namespace is not likely to consolidate many identical assets, so it doesn't really achieve much other than simplicity. The only exception I can think of is the case where an instance has course(s) that are white-labelled and offered by many different orgs on the same instance. In that one case, a global namespace would be a big win.

(Technically it could be achieved via some sort of "use global namespace" vs. "use org namepsace" vs. "use course-specific namespace" setting, but that seems like a configuration setting that's not worth the complexity it entails. And by the time any particular instance realizes they may want to change that setting, it's likely too late anyways.)

So I think we should pick whichever one we feel is the best overall balance.

[ormsbee]

@kdmccormick:

Confirming my understanding: We would have a site-wide namespace if not for the possibility of a birthday attack, right?

I agree with the points that @bradenmacdonald made. In practice, almost all sharing of content happens within an org today, so we don't lose much. And in addition to potential long term security issues, there are also cost and potentially performance issues:

1. Having an org-level separation makes it more obvious to operators when a certain org is using a disproportionate amount of resources.
2. It also guards against potential performance issues where we start getting bad behavior because the backing filesystem can't gracefully handle a really large directory. I don't think this will be an issue with modern filesystems, but keeping them separate both makes the directories smaller and gives a little more isolation so that one really bloated org media dir won't necessarily slow things down for everyone else. (Even if the fs itself is performant, there can be various libraries/utilities that can break down in those circumstances.)

[bradenmacdonald]

I would like to be in a place where someday where we can export a self-contained LearningPackage that contains a Pathway, its half dozen courses, and a bunch of smaller things

@ormsbee I'm not opposed to this idea by any means, and I'll try to incorporate that as a future option into the ADR. But I don't really grasp how it will be useful. If LearningPackages are a "behind the scenes" thing, how and why would users even do this? Do we have any requests for this from users/product teams? Or is this sort of package a developer-oriented thing that is created and imported to various instances?

I do think that we should leave the door open for scenarios where there is a team working on content in a more centralized way, and it makes sense for that team to have such broad permissions across multiple courses or smaller learning contexts (e.g. bootcamps).

Definitely, but I don't think LearningPackage is a user-visible permissions boundary, is it? Permissions are usually set at the LearningContext level instead. The authz system should allow creating arbitrary groups of courses (e.g. all courses that match pattern BOOTCAMP5_* in their course code) and assigning permissions that way to groups of users.

[bradenmacdonald]

Maybe this means that we'll need to make a new model to capture namespaced identifiers to resolve potential conflicts between different course runs, but I think this sort of thing is an acceptable trade-off.

Could this be the Usage table?

It could, but that would only namespace Components. Everything else in the course that may exist as versioned content (static assets, pages, grading policy, etc.) would also need some way of being associated with a specific run.

In #360 there was some discussion about making "Files & Uploads" and other content part of the hierarchy. In that case, a single table could be used both to represent the complete content hierarchy and to provide namespacing of component/container codes. But it seems weird to me to make everything part of the outline, when the learners don't see it that way.

I will try to incorporate this option into the ADR, but I'm leaning toward sketching it out in detail as a "rejected alternative" that we can consider in the future.

[bradenmacdonald]

I have made some updates to the ADR:

• Renamed blob_namespace -> media_file_namespace
• Change to de-duplicate assets at the per-org level going forward, still using the media_file_namespace concept though.
• Added a big new "rejected" (but planned for the future) section for flexibly combining runs and more into shared learning packages

[bradenmacdonald]

contrary to the existing assumption that these identifiers are opaque

I had some recollection that OEP-68 or ADR 0003 recommended that "codes should not be parsed" (e.g. avoid inventing custom prefixing schemes in "code" fields), but I can't find that now.

[ormsbee]

One thing we could consider for re-runs is to have optional fields on PublishableEntity that capture another entity and version that it was derived from. So when a re-run happens, we could create complete copies of everything, but basically say, "Component C2 in CourseRun2 was derived from Component C1 in CourseRun1, and the draft version of C1 was v10 at the time C2 was created." If we decided to support multiple runs in the same LearningPackage, it could be a lightweight way of tracking/differentiating what would otherwise be confusing duplicate data.

Or I guess we could also re-purpose copied_from_block, though I feel like this is a bit different.

In any case, none of this contradicts this ADR. It just came to mind when considering the idea of links.

[bradenmacdonald]

Yes, we already have copied_from_block and ComponentLink, so I do think there's plenty of precedent for models that track where the copies came from. That's also what I was trying to get at with this part of the ADR:

There are other ways to track a component’s history. Between DraftChangeLog, PublishLog, openedx-platform’s ComponentLink, and potential future RerunLog models, we can find ways to show a component’s complete provenance history without necessarily implementing copy-on-write nor completely duplicating it.

[ormsbee]

@ormsbee I'm not opposed to this idea by any means, and I'll try to incorporate that as a future option into the ADR. But I don't really grasp how it will be useful. If LearningPackages are a "behind the scenes" thing, how and why would users even do this? Do we have any requests for this from users/product teams? Or is this sort of package a developer-oriented thing that is created and imported to various instances?

@bradenmacdonald: I admit that it's still a bit fuzzy in my head, but I think of LearningPackages as the authoring equivalent of a learning context, i.e. the standard owned thing that can be authored, imported, and exported. Maybe that one thing is a library that has a bunch of assorted content in it. Or maybe that shared, authorable thingy has a bunch of course runs. Or maybe all LearningPackages are Libraries, and we just happen to put various combinations of larger things in those Libraries. But I like the idea of the bounds of the LearningPackage being whatever is convenient and useful for the purposes of authoring. So if it makes sense for one team to own a bootcamp pathway and a bunch of mini-courselets that are associated with that bootcamp, then they should be able to operate on it in a single place.

[ormsbee]

One of the reasons I assume that the LearningPackage is going to be the boundary for this is that our content is so interconnected that doing permissions on only subsets of content within a LearningPackage is going to be a real pain. In my mind, different authoring ownership == different LearningPackage + sync mechanism if they need to collaborate.

[ormsbee]

Maybe this means that we'll need to make a new model to capture namespaced identifiers to resolve potential conflicts between different course runs, but I think this sort of thing is an acceptable trade-off.

Could this be the Usage table?

It could, but that would only namespace Components. Everything else in the course that may exist as versioned content (static assets, pages, grading policy, etc.) would also need some way of being associated with a specific run.

Is this something that we could actually leverage Collections for? Introduce a new, nullable field to CollectionPublishableEntity that gives a Collection-local identifier? Entities still need a LearningPackage-scoped ref, but any custom mapping/conflict resolution could happen via this new field? We already import courses into their own Collection anyhow.

[ormsbee]

I suppose at that point, the confusing part would be when to use which identifier, particularly if we think there would be fallback rules. Still, I think there's something there we can use without it turning into a giant mess.

[ormsbee]

Riffing on the Collection thing a bit more—in the longer run, we could introduce Collection types, one of which is a CourseRunCollection. Each Collection type could be associated with its own libraries detail view.

One place where there's a mismatch in semantics is that creating child elements for something in a Collection doesn't automatically add those things to that Collection. I'm not sure if that would be a feature or a bug in this case...

[bradenmacdonald]

@ormsbee

One of the reasons I assume that the LearningPackage is going to be the boundary for this is that our content is so interconnected that doing permissions on only subsets of content within a LearningPackage is going to be a real pain. In my mind, different authoring ownership == different LearningPackage + sync mechanism if they need to collaborate.

If we're going to be designing around that idea, then I'd really like to see a product definition, and at least some sort of minimal UX plan and input from the authz team. Otherwise, I think the idea is too vague to be useful for architecture planning purposes. I'm also wary of the many times I've personally built out a backend feature that never became useable by actual users (e.g. pinned versions of containers).

[bradenmacdonald]

@ormsbee I like the collections idea and have included a mention of it in the ADR now, 61baea9

[bradenmacdonald]

(rebased without changes)

[ormsbee]

I'm honestly still not 100% sold on making CourseRun and LearningPackage 1:1. I prefer the idea that the PublishableEntity refs are an authoring convention, but that all identifiers for learning contexts would be through some separate lookup mapping (including UsageKeys for blocks in courses). I could imagine similar situations coming up for expressing multiple LTI context mappings in a LearningPackage, or serving a standalone Section through the LMS. I feel like I should be able to have any combination of content in a LearningPackage, and have permissions on that package that are independent of the execution permissions on the things inside of it.

That being said, I get that what is being proposed here is a simpler starting position to go with. I also think that we can add another layer of namespaced identifiers to map multiple CourseRuns into the same LearningPackage later if need-be, even if maybe only Libraries support that feature.

I'm left with this nagging feeling that we're missing a simple abstraction that would make this straightforward, but I'm not going to hold up this PR over it.

Thank you.

[kdmccormick]

After thinking it through again, I'd also rather avoid locking into a 1:1 mapping. Even if we never allow multiple CourseRuns in one LearningPackage, I want to make sure that we could have each of these be a LearningContext without requiring them to have a dedicated LearningPackage:

• Standalone enrollable sections, subsections, units
• Library items as an LTI tool
• CCX, rebuilt more simply on top of openedx-core

Let's not worry about fully planning out a model that would allow multiple CourseRuns to exist in a LearningPackage, but I think we should avoid baking in the assumption that a LearningContext is 1:1 with a LearningPackage, or that a Usage is 1:1 with a PublishableEntity. I think developers would be very quick to make this assumption if we let them, and it'd be a nightmare to undo.

I'm still trying to sketch out what the "missing simple abstraction" Dave mentions could be, but even without that new abstraction, I'd be happier to merge this ADR if:

1. we drop the assumption that package_ref matches the course_key. I'd rather it explicitly just be a convention.
2. we state that usage keys are not to be derived directly from the learning package-local refs and codes. Instead, there should be some API function like get_entity_usage_key(learning_context, publishable_entity) which owns the mapping, allowing us to add logic for standalone containers, LTI, and CCX later on.
3. we allow multiple CourseContents to point at the same LearningPackage. We don't need to follow all the implications through--just have the data model permit it, and have a unit test for it.

@ormsbee @bradenmacdonald what do you think?

[bradenmacdonald]

@kdmccormick @ormsbee

• Standalone enrollable sections, subsections, units
• Library items as an LTI tool
• CCX, rebuilt more simply on top of openedx-core

Here you are talking about different learning contexts being different views of the same content in the same learning package. I think that's a very reasonable use case, and something we more or less have today in some way already.

But the other idea that has come up is to have diverging/unrelated content in the same learning package - multiple runs of a course, or a course and semi-related micro-course, or a course and a library and a pathway. And that's a completely different thing - suddenly, we don't just have to worry about mapping codes to usage keys, but we have all kinds of issues with scoping and code (component_code) conflicts. That's the part that I haven't yet seen a product justification for.

If you read the ADRs in #822 we already have to decide now if a learning package is a single namespace or not. If we can say it is, it simplifies a lot. On the other hand, if we say that we need to allow completely unrelated assets to share the same component code in the same learning package, it's going to require a lot more models and mapping than what I've sketched out so far, and we're already creating a ton of new rows in the DB even without that.

So yeah, I guess I'm agreeing with @kdmccormick's point here.

[bradenmacdonald]

we drop the assumption that package_ref matches the course_key. I'd rather it explicitly just be a convention.

What do you think about setting package_ref to course_key + short_random_string? If we just set package_ref = course_key, at some point someone is going to assume they're the same thing and write code that depends on it.

• we state that usage keys are not to be derived directly from the learning package-local refs and codes. Instead, there should be some API function like get_entity_usage_key(learning_context, publishable_entity) which owns the mapping, allowing us to add logic for standalone containers, LTI, and CCX later on.

This would have to go along with a get_contexts_for_package(package) API, and would naturally fit well in some module that also offered a get_entity(context, usage_key) API.

Most of our lookups go from usage key to entity, and not the other way around, but the main exception to this would be our publishing events. When entities are modified in Studio in any learning package (library or in future courses), we send out ENTITIES_DRAFT_CHANGED and ENTITIES_PUBLISHED events which have only the learning_package and entity IDs. Currently, our search index stores those things by learning context though (we should change that so studio stores them by package - that would simplify a lot!). For the learner-facing search though, a handler would need to listen to those events, and then do a fan out: get all the learning contexts associate with the package, and compute all the usage keys for all the entities across all those contexts, and then update the learner-facing search index, which will definitely be context based.

[bradenmacdonald]

(thinking a bit more) I do have the feeling that there's still a larger product conversation that needs to take place here, even with the more limited (different views of same content) approach.

If we refactor the studio-side search index to be based around learning packages instead of contexts (which isn't strictly necessary but would make a lot of sense), then the search query permissions would probably need to be refactored to be based around learning packages, not learning contexts.

As soon as we have multiple contexts sharing the same package, you have a situation where changes to content in one context affect another context. So we either need to have rules like "standalone sections and CCX courses are strictly read-only and authors can't edit them" OR we need to grant permissions at the learning package level OR some complicated hybrid.

So my feeling here is that "learning package" may need to become an author-visible concept and also a permissions boundary, but I don't think we have any product definition nor is the authz team working on that at all. So if my feeling here makes sense, we should try to get buy-in from them, though it doesn't have to block this ADR.

[bradenmacdonald]

we should avoid baking in the assumption that a LearningContext is 1:1 with a LearningPackage, or that a Usage is 1:1 with a PublishableEntity. I think developers would be very quick to make this assumption if we let them, and it'd be a nightmare to undo.

I'm still trying to sketch out what the "missing simple abstraction" Dave mentions could be, but even without that new abstraction, I'd be happier to merge this ADR if:

1. we drop the assumption that package_ref matches the course_key. I'd rather it explicitly just be a convention.
2. we state that usage keys are not to be derived directly from the learning package-local refs and codes. Instead, there should be some API function like get_entity_usage_key(learning_context, publishable_entity) which owns the mapping, allowing us to add logic for standalone containers, LTI, and CCX later on.
3. we allow multiple CourseContents to point at the same LearningPackage. We don't need to follow all the implications through--just have the data model permit it, and have a unit test for it.

@kdmccormick @ormsbee OK, I have pushed 87a3fa1 to enshrine these principles. Please take a look.

[kdmccormick]

If we refactor the studio-side search index to be based around learning packages instead of contexts (which isn't strictly necessary but would make a lot of sense), then the search query permissions would probably need to be refactored to be based around learning packages, not learning contexts.

I think this is a good idea

As soon as we have multiple contexts sharing the same package, you have a situation where changes to content in one context affect another context. So we either need to have rules like "standalone sections and CCX courses are strictly read-only and authors can't edit them" OR we need to grant permissions at the learning package level OR some complicated hybrid.

Where I'd like to eventually get to is that:

• learning/instruction permission are granted only on the learning context
• authoring permissions are granted only on the learning package
• we would tie these layers together using roles. for example;
  • the CourseStaff role would grant write permission on both the course's learning context and the course's underlying learning package.
  • the CCXCoach role would grant write permission on the CCX learning context, but only grant read on the underlying learning package.
  • the Library roles would grant read and write on both the underlying learning package and the library's preview-only learning context.

(I'm using "read" and "write" loosely here, but I think you know what I mean)

You're right that we need product input and the authz maintainers' buy-in on that--I can work on shopping this around to them. In the meantime, I think we can get away with keeping keeping permission checks at the course/library API level, and not making permission checks against learning packages directly. Do you agree or do you feel like it'll get messy quickly if we don't sort this out soon?

[kdmccormick]

thank you, looks great! all my remaining comments are non-blocking, merge as you see fit.

[kdmccormick]

I like explicitly decoupling the package_ref from the course_key like this 👍🏻

One alternate idea for deriving package_ref: take the course key, and turn it into shell-safe slug like course_MITx_Math100_2026Fall that (one day in the future) could be the used as the default filename of the package backup (course_MITx_Math100_2026Fall.zip). I have no justification for this other than "seems nice", so up to you if you want to use this, or keep your current suggestion.

[bradenmacdonald]

I added this suggestion as an example, but postponed the final decision until implementation.

[kdmccormick]

I wonder if it'd be better to call this something more on-the-nose, like CourseRunLearningPackage or LearningPackageForCourseRun.

not a strong opinion -- the current name is fine, too.

[bradenmacdonald]

I guess it depends if course_run and learning_package are the only two fields, or if there's any other associated metadata. If it's just those two fields, your suggestion is better. I've added it into the text but stuck with CourseContent tentatively.

[bradenmacdonald]

You're right that we need product input and the authz maintainers' buy-in on that--I can work on shopping this around to them. In the meantime, I think we can get away with keeping keeping permission checks at the course/library API level, and not making permission checks against learning packages directly. Do you agree or do you feel like it'll get messy quickly if we don't sort this out soon?

@kdmccormick Yeah, I think we're OK for now, but we should figure it out before we get to actually having multiple contexts per package, e.g. when we support CCX. And thanks for offering to shop that around. The earlier we can get buy-in, the better, since learning package is more or less a whole new paradigm for Studio that we haven't had yet on the user side.