Skip to content

Commit dc73edc

Browse files
committed
Add tooling to process site packages in an Apple Xcode project.
1 parent 3ad9316 commit dc73edc

10 files changed

Lines changed: 84 additions & 110 deletions

File tree

‎Doc/using/ios.rst‎

Lines changed: 24 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -194,13 +194,21 @@ To add Python to an iOS Xcode project:
194194
adjusting paths as needed. Ensure that this folder is associated with your
195195
app target.
196196

197-
4. Select the app target by selecting the root node of your Xcode project, then
197+
4. Create folders to store your third-party dependencies (e.g., packages you've
198+
installed from PyPI). You'll need a separate dependencies folder for iOS
199+
devices and iOS simulators. In the following instructions, we'll assume
200+
you've used folders named ``app_packages.iphoneos`` and
201+
``app_packages.iphonesimulator``, located in the root of your project; you
202+
can use any other location by adjusting paths as needed. These folders
203+
*should not* be associated with the app target.
204+
205+
5. Select the app target by selecting the root node of your Xcode project, then
198206
the target name in the sidebar that appears.
199207

200-
5. In the "General" settings, under "Frameworks, Libraries and Embedded
208+
6. In the "General" settings, under "Frameworks, Libraries and Embedded
201209
Content", add ``Python.xcframework``, with "Embed & Sign" selected.
202210

203-
6. In the "Build Settings" tab, modify the following:
211+
7. In the "Build Settings" tab, modify the following:
204212

205213
- Build Options
206214

@@ -216,7 +224,7 @@ To add Python to an iOS Xcode project:
216224

217225
* Quoted Include In Framework Header: No
218226

219-
7. Add a build step that processes the Python standard library, and your own
227+
8. Add a build step that processes the Python standard library, and your own
220228
Python binary dependencies. In the "Build Phases" tab, add a new "Run
221229
Script" build step *before* the "Embed Frameworks" step, but *after* the
222230
"Copy Bundle Resources" step. Name the step "Process Python libraries",
@@ -228,11 +236,18 @@ To add Python to an iOS Xcode project:
228236
set -e
229237
source "$PROJECT_DIR/Python.xcframework/build/utils.sh"
230238
install_python Python.xcframework app
239+
if [ "$EFFECTIVE_PLATFORM_NAME" = "-iphonesimulator" ]; then
240+
APP_PACKAGES_DIR="$PROJECT_DIR/app_packages.iphonesimulator/"
241+
else
242+
APP_PACKAGES_DIR="$PROJECT_DIR/app_packages.iphoneos/"
243+
fi
244+
rsync -au --delete "$APP_PACKAGES_DIR" "$CODESIGNING_FOLDER_PATH/app_packages"
245+
install_site_packages Python.xcframework app_packages
231246
232247
If you have placed your XCframework somewhere other than the root of your
233248
project, modify the path to the first argument.
234249

235-
8. Add Objective C code to initialize and use a Python interpreter in embedded
250+
9. Add Objective C code to initialize and use a Python interpreter in embedded
236251
mode. You should ensure that:
237252

238253
* UTF-8 mode (:c:member:`PyPreConfig.utf8_mode`) is *enabled*;
@@ -241,36 +256,14 @@ To add Python to an iOS Xcode project:
241256
* Signal handlers (:c:member:`PyConfig.install_signal_handlers`) are *enabled*;
242257
* System logging (:c:member:`PyConfig.use_system_logger`) is *enabled*
243258
(optional, but strongly recommended; this is enabled by default);
244-
* :envvar:`PYTHONHOME` for the interpreter is configured to point at the
245-
``python`` subfolder of your app's bundle; and
246-
* The :envvar:`PYTHONPATH` for the interpreter includes:
247-
248-
- the ``python/lib/python3.X`` subfolder of your app's bundle,
249-
- the ``python/lib/python3.X/lib-dynload`` subfolder of your app's bundle, and
250-
- the ``app`` subfolder of your app's bundle
259+
* The interpreter's home direectory (:c:member:`PyConfig.home`) is configured
260+
to point at the ``python`` subfolder of your app's bundle; and
261+
* The interpreter's module search paths (:c:member:`PyConfig.home`) includes
262+
the ``app`` subfolder of your app's bundle.
251263

252264
Your app's bundle location can be determined using ``[[NSBundle mainBundle]
253265
resourcePath]``.
254266

255-
Steps 7 and 8 of these instructions assume that you have a single folder of
256-
pure Python application code, named ``app``. If you have third-party binary
257-
modules in your app, some additional steps will be required:
258-
259-
* You need to ensure that any folders containing third-party binaries are
260-
either associated with the app target, or are explicitly copied as part of
261-
step 7. Step 7 should also purge any binaries that are not appropriate for
262-
the platform a specific build is targeting (i.e., delete any device binaries
263-
if you're building an app targeting the simulator).
264-
265-
* If you're using a separate folder for third-party packages, ensure that
266-
folder is added to the end of the call to ``install_python`` in step 7, and
267-
as part of the :envvar:`PYTHONPATH` configuration in step 8.
268-
269-
* If any of the folders that contain third-party packages will contain ``.pth``
270-
files, you should add that folder as a *site directory* (using
271-
:meth:`site.addsitedir`), rather than adding to :envvar:`PYTHONPATH` or
272-
:attr:`sys.path` directly.
273-
274267
Testing a Python package
275268
------------------------
276269

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
The iOS build utility script now contains better support for integrating
2+
third-party code into a project's site-packages folder. The testbed has
3+
been modified to use this script, ensuring that sub-interpreters can find
4+
all installed third-party dependencies.

‎Platforms/Apple/testbed/Python.xcframework/build/utils.sh‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,8 @@
1515
# -----
1616
# set -e
1717
# source "$PROJECT_DIR/Python.xcframework/build/utils.sh"
18-
# install_python Python.xcframework app app_packages
18+
# install_python Python.xcframework app
19+
# install_site_packages Python.xcframework app_packages
1920
# -----
2021

2122
# Copy the standard library from the XCframework into the app bundle.
@@ -161,20 +162,39 @@ process_dylibs () {
161162
#
162163
# Accepts 1 or more arguments:
163164
# 1. The path, relative to the root of the Xcode project, where the Python
164-
# XCframework can be found. If the XCframework is in the root of the project,
165-
# 2+. The path of a package, relative to the root of the packaged app, that contains
166-
# library content that should be processed for binary libraries.
165+
# XCframework can be found.
166+
# 2+. The path of any package, relative to the root of the Xcode project, that contains
167+
# user-space Python code that should be processed for binary libraries.
167168
install_python() {
168169
PYTHON_XCFRAMEWORK_PATH=$1
169170
shift
170171

171172
install_stdlib "$PYTHON_XCFRAMEWORK_PATH"
172-
PYTHON_VER=$(ls -1 "$CODESIGNING_FOLDER_PATH/python/lib" | grep -E "^python3\.\d+[a-z]*$")
173-
echo "Install Python $PYTHON_VER standard library extension modules..."
174-
process_dylibs "$PYTHON_XCFRAMEWORK_PATH" python/lib/$PYTHON_VER/lib-dynload
173+
PYTHON_LIBDIR=$(ls -1 "$CODESIGNING_FOLDER_PATH/python/lib" | grep -E "^python3\.\d+[a-z]*$")
174+
175+
echo "Install Python $PYTHON_LIBDIR standard library extension modules..."
176+
process_dylibs "$PYTHON_XCFRAMEWORK_PATH" python/lib/$PYTHON_LIBDIR/lib-dynload
175177

176178
for package_path in "$@"; do
177179
echo "Installing $package_path extension modules ..."
178180
process_dylibs "$PYTHON_XCFRAMEWORK_PATH" "$package_path"
179181
done
180182
}
183+
184+
# The entry point for post-processing the site packages for a Python XCframework.
185+
#
186+
# Accepts 2 arguments:
187+
# 1. The path, relative to the root of the Xcode project, where the Python
188+
# XCframework can be found.
189+
# 2. The path, relative to the root of the Xcode project, that contains
190+
# the site-packages content for the app.
191+
install_site_packages() {
192+
PYTHON_XCFRAMEWORK_PATH=$1
193+
APP_PACKAGES_PATH=$2
194+
PYTHON_LIBDIR=$(ls -1 "$CODESIGNING_FOLDER_PATH/python/lib" | grep -E "^python3\.\d+[a-z]*$")
195+
196+
echo "Install Python $PYTHON_LIBDIR site-packages extension modules..."
197+
rm -rf "$CODESIGNING_FOLDER_PATH/python/lib/$PYTHON_LIBDIR/site-packages"
198+
mv "$CODESIGNING_FOLDER_PATH/$APP_PACKAGES_PATH" "$CODESIGNING_FOLDER_PATH/python/lib/$PYTHON_LIBDIR/site-packages"
199+
process_dylibs "$PYTHON_XCFRAMEWORK_PATH" python/lib/$PYTHON_LIBDIR/site-packages
200+
}

‎Platforms/Apple/testbed/TestbedTests/TestbedTests.m‎

Lines changed: 10 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -11,17 +11,9 @@ @implementation TestbedTests
1111
- (void)testPython {
1212
const char **argv;
1313
int exit_code;
14-
int failed;
1514
PyStatus status;
1615
PyPreConfig preconfig;
1716
PyConfig config;
18-
PyObject *app_packages_path;
19-
PyObject *method_args;
20-
PyObject *result;
21-
PyObject *site_module;
22-
PyObject *site_addsitedir_attr;
23-
PyObject *sys_module;
24-
PyObject *sys_path_attr;
2517
NSArray *test_args;
2618
NSString *python_home;
2719
NSString *path;
@@ -115,73 +107,24 @@ - (void)testPython {
115107
return;
116108
}
117109

118-
NSLog(@"Initializing Python runtime...");
119-
status = Py_InitializeFromConfig(&config);
120-
if (PyStatus_Exception(status)) {
121-
XCTFail(@"Unable to initialize Python interpreter: %s", status.err_msg);
122-
PyConfig_Clear(&config);
123-
return;
124-
}
125-
126-
// Add app_packages as a site directory. This both adds to sys.path,
127-
// and ensures that any .pth files in that directory will be executed.
128-
site_module = PyImport_ImportModule("site");
129-
if (site_module == NULL) {
130-
XCTFail(@"Could not import site module");
131-
return;
132-
}
133-
134-
site_addsitedir_attr = PyObject_GetAttrString(site_module, "addsitedir");
135-
if (site_addsitedir_attr == NULL || !PyCallable_Check(site_addsitedir_attr)) {
136-
XCTFail(@"Could not access site.addsitedir");
137-
return;
138-
}
139-
140-
path = [NSString stringWithFormat:@"%@/app_packages", resourcePath, nil];
141-
NSLog(@"App packages path: %@", path);
142-
wtmp_str = Py_DecodeLocale([path UTF8String], NULL);
143-
app_packages_path = PyUnicode_FromWideChar(wtmp_str, wcslen(wtmp_str));
144-
if (app_packages_path == NULL) {
145-
XCTFail(@"Could not convert app_packages path to unicode");
146-
return;
147-
}
148-
PyMem_RawFree(wtmp_str);
149-
150-
method_args = Py_BuildValue("(O)", app_packages_path);
151-
if (method_args == NULL) {
152-
XCTFail(@"Could not create arguments for site.addsitedir");
153-
return;
154-
}
155-
156-
result = PyObject_CallObject(site_addsitedir_attr, method_args);
157-
if (result == NULL) {
158-
XCTFail(@"Could not add app_packages directory using site.addsitedir");
159-
return;
160-
}
161-
162-
// Add test code to sys.path
163-
sys_module = PyImport_ImportModule("sys");
164-
if (sys_module == NULL) {
165-
XCTFail(@"Could not import sys module");
166-
return;
167-
}
168-
169-
sys_path_attr = PyObject_GetAttrString(sys_module, "path");
170-
if (sys_path_attr == NULL) {
171-
XCTFail(@"Could not access sys.path");
172-
return;
173-
}
174-
175110
path = [NSString stringWithFormat:@"%@/app", resourcePath, nil];
176111
NSLog(@"App path: %@", path);
177112
wtmp_str = Py_DecodeLocale([path UTF8String], NULL);
178-
failed = PyList_Insert(sys_path_attr, 0, PyUnicode_FromString([path UTF8String]));
179-
if (failed) {
113+
status = PyWideStringList_Insert(&config.module_search_paths, 0, wtmp_str);
114+
if (PyStatus_Exception(status)) {
180115
XCTFail(@"Unable to add app to sys.path");
181116
return;
182117
}
183118
PyMem_RawFree(wtmp_str);
184119

120+
NSLog(@"Initializing Python runtime...");
121+
status = Py_InitializeFromConfig(&config);
122+
if (PyStatus_Exception(status)) {
123+
XCTFail(@"Unable to initialize Python interpreter: %s", status.err_msg);
124+
PyConfig_Clear(&config);
125+
return;
126+
}
127+
185128
// Ensure the working directory is the app folder.
186129
chdir([path UTF8String]);
187130

‎Platforms/Apple/testbed/__main__.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -241,7 +241,7 @@ def expand_template(location: Path, abiflags: str) -> None:
241241
replacement = abiflags.upper()
242242
for path in location.rglob("*"):
243243
if (
244-
path.suffix in {".m", ".pbxproj", ".plist"}
244+
path.suffix in {".m", ".pbxproj", ".plist", ".sh"}
245245
and path.is_file()
246246
and not path.is_symlink()
247247
):
@@ -330,6 +330,7 @@ def clone_testbed(
330330
"__main__.py",
331331
"TestbedTests",
332332
"Testbed.lldbinit",
333+
"prepare_python_code.sh",
333334
f"{platform}Testbed",
334335
f"{platform}Testbed.xcodeproj",
335336
f"{platform}Testbed.xctestplan",

‎Platforms/Apple/testbed/iOSTestbed.xcodeproj/project.pbxproj‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
/* End PBXCopyFilesBuildPhase section */
5050

5151
/* Begin PBXFileReference section */
52+
60214C9030AD4273006FA640 /* prepare_python_code.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = prepare_python_code.sh; sourceTree = "<group>"; };
5253
607A66122B0EFA380010BFC8 /* iOSTestbed.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = iOSTestbed.app; sourceTree = BUILT_PRODUCTS_DIR; };
5354
607A662D2B0EFA3A0010BFC8 /* iOSTestbedTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = iOSTestbedTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
5455
607A664A2B0EFB310010BFC8 /* Python{{abiflags}}.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; path = "Python{{abiflags}}.xcframework"; sourceTree = "<group>"; };
@@ -110,6 +111,7 @@
110111
children = (
111112
60FE0EFB2E56BB6D00524F87 /* iOSTestbed.xctestplan */,
112113
607A664A2B0EFB310010BFC8 /* Python{{abiflags}}.xcframework */,
114+
60214C9030AD4273006FA640 /* prepare_python_code.sh */,
113115
607A66142B0EFA380010BFC8 /* iOSTestbed */,
114116
607A66302B0EFA3A0010BFC8 /* TestbedTests */,
115117
607A66132B0EFA380010BFC8 /* Products */,
@@ -250,7 +252,7 @@
250252
);
251253
runOnlyForDeploymentPostprocessing = 0;
252254
shellPath = /bin/sh;
253-
shellScript = "set -e\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\ninstall_python Python{{abiflags}}.xcframework app app_packages\n";
255+
shellScript = "set -e\nsource \"$PROJECT_DIR/prepare_python_code.sh\"\n";
254256
showEnvVarsInLog = 0;
255257
};
256258
/* End PBXShellScriptBuildPhase section */
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/sh
2+
set -e
3+
source "$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh"
4+
install_python "Python{{abiflags}}.xcframework" app
5+
install_site_packages "Python{{abiflags}}.xcframework" app_packages

‎Platforms/Apple/testbed/tvOSTestbed.xcodeproj/project.pbxproj‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
/* End PBXCopyFilesBuildPhase section */
3838

3939
/* Begin PBXFileReference section */
40+
60214C9030AD4273006FA640 /* prepare_python_code.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = prepare_python_code.sh; sourceTree = "<group>"; };
4041
6077B3802E82A4BE00E3D6A3 /* tvOSTestbed.xctestplan */ = {isa = PBXFileReference; lastKnownFileType = text; path = tvOSTestbed.xctestplan; sourceTree = "<group>"; };
4142
EE7C8A1C2DCD6FF3003206DB /* Python{{abiflags}}.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; path = Python{{abiflags}}.xcframework; sourceTree = "<group>"; };
4243
EE989E4E2DCD6E780036B268 /* tvOSTestbed.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = tvOSTestbed.app; sourceTree = BUILT_PRODUCTS_DIR; };
@@ -98,6 +99,7 @@
9899
children = (
99100
6077B3802E82A4BE00E3D6A3 /* tvOSTestbed.xctestplan */,
100101
EE7C8A1C2DCD6FF3003206DB /* Python{{abiflags}}.xcframework */,
102+
60214C9030AD4273006FA640 /* prepare_python_code.sh */,
101103
EE989E502DCD6E780036B268 /* tvOSTestbed */,
102104
EE989E682DCD6E7A0036B268 /* TestbedTests */,
103105
EE989E4F2DCD6E780036B268 /* Products */,
@@ -236,7 +238,7 @@
236238
);
237239
runOnlyForDeploymentPostprocessing = 0;
238240
shellPath = /bin/sh;
239-
shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n";
241+
shellScript = "set -e\nsource \"$PROJECT_DIR/prepare_python_code.sh\"\n";
240242
showEnvVarsInLog = 0;
241243
};
242244
/* End PBXShellScriptBuildPhase section */

‎Platforms/Apple/testbed/visionOSTestbed.xcodeproj/project.pbxproj‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
/* End PBXCopyFilesBuildPhase section */
5050

5151
/* Begin PBXFileReference section */
52+
60214C9030AD4273006FA640 /* prepare_python_code.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = prepare_python_code.sh; sourceTree = "<group>"; };
5253
6077B3D62E82E60C00E3D6A3 /* visionOSTestbed.xctestplan */ = {isa = PBXFileReference; lastKnownFileType = text; path = visionOSTestbed.xctestplan; sourceTree = "<group>"; };
5354
607A66122B0EFA380010BFC8 /* visionOSTestbed.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = visionOSTestbed.app; sourceTree = BUILT_PRODUCTS_DIR; };
5455
607A662D2B0EFA3A0010BFC8 /* TestbedTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = TestbedTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
@@ -110,6 +111,7 @@
110111
children = (
111112
6077B3D62E82E60C00E3D6A3 /* visionOSTestbed.xctestplan */,
112113
EEE9C80C2DAB5ECA0056F8C6 /* Python{{abiflags}}.xcframework */,
114+
60214C9030AD4273006FA640 /* prepare_python_code.sh */,
113115
607A66142B0EFA380010BFC8 /* visionOSTestbed */,
114116
607A66302B0EFA3A0010BFC8 /* TestbedTests */,
115117
607A66132B0EFA380010BFC8 /* Products */,
@@ -250,7 +252,7 @@
250252
);
251253
runOnlyForDeploymentPostprocessing = 0;
252254
shellPath = /bin/sh;
253-
shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n";
255+
shellScript = "set -e\nsource \"$PROJECT_DIR/prepare_python_code.sh\"\n";
254256
showEnvVarsInLog = 0;
255257
};
256258
/* End PBXShellScriptBuildPhase section */

‎Platforms/Apple/testbed/watchOSTestbed.xcodeproj/project.pbxproj‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
/* End PBXCopyFilesBuildPhase section */
3838

3939
/* Begin PBXFileReference section */
40+
60214C9030AD4273006FA640 /* prepare_python_code.sh */ = {isa = PBXFileReference; lastKnownFileType = text.script.sh; path = prepare_python_code.sh; sourceTree = "<group>"; };
4041
60F7C0062EB0000000A1B2C3 /* watchOSTestbed.xctestplan */ = {isa = PBXFileReference; lastKnownFileType = text; path = watchOSTestbed.xctestplan; sourceTree = "<group>"; };
4142
60F7C0072EB0000000A1B2C3 /* Python{{abiflags}}.xcframework */ = {isa = PBXFileReference; lastKnownFileType = wrapper.xcframework; path = Python{{abiflags}}.xcframework; sourceTree = "<group>"; };
4243
60F7C0082EB0000000A1B2C3 /* watchOSTestbed.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = watchOSTestbed.app; sourceTree = BUILT_PRODUCTS_DIR; };
@@ -98,6 +99,7 @@
9899
children = (
99100
60F7C0062EB0000000A1B2C3 /* watchOSTestbed.xctestplan */,
100101
60F7C0072EB0000000A1B2C3 /* Python{{abiflags}}.xcframework */,
102+
60214C9030AD4273006FA640 /* prepare_python_code.sh */,
101103
60F7C00B2EB0000000A1B2C3 /* watchOSTestbed */,
102104
60F7C00C2EB0000000A1B2C3 /* TestbedTests */,
103105
60F7C0102EB0000000A1B2C3 /* Products */,
@@ -236,7 +238,7 @@
236238
);
237239
runOnlyForDeploymentPostprocessing = 0;
238240
shellPath = /bin/sh;
239-
shellScript = "set -e\n\nsource \"$PROJECT_DIR/Python{{abiflags}}.xcframework/build/utils.sh\"\n\ninstall_python Python{{abiflags}}.xcframework app app_packages\n";
241+
shellScript = "set -e\nsource \"$PROJECT_DIR/prepare_python_code.sh\"\n";
240242
showEnvVarsInLog = 0;
241243
};
242244
/* End PBXShellScriptBuildPhase section */

0 commit comments

Comments
 (0)