Commit 9c93dc34dee1be9db10bad8208a4798fe1cd0b83 - python-magic-ahupp

+10

-0

.gitignore less more

	0	.coverage*
	1	.tox/
	2	bin/
0	3	deb_dist
	4	htmlcov/
	5	lib/
	6	__pycache__/
1	7	python_magic.egg-info
	8	pip-selfcheck.json
	9	pyvenv.cfg
	10	*.pyc
	11	*~

+9

-15

.travis.yml less more

0	0	language: python
1
2		# needed to use trusty
3		sudo: required
4
5		dist: trusty
	1	dist: xenial
	2	cache: pip
6	3
7	4	python:
8		- "2.6"
9	5	- "2.7"
10		- "3.3"
11		- "3.4"
12	6	- "3.5"
13	7	- "3.6"
14		- "nightly"
	8	- "3.7"
	9	- "3.8"
15	10
16	11	install:
17		- pip install coveralls
18		- pip install codecov
19		- python setup.py install
	12	- pip install coverage
	13	- pip install .
20	14
21	15	script:
22		- coverage run setup.py test
	16	- LC_ALL=en_US.UTF-8 coverage run ./test/test.py
23	17
24	18	after_success:
25		- coveralls
26		- codecov
	19	- pip install coveralls && coveralls
	20	- pip install codecov && codecov

+32

-0

CHANGELOG less more

	0	Changes in 0.4.18
	1
	2	- Make bindings for magic_[set\|get]param optional, and throw NotImplementedError
	3	if they are used but not supported. Only call setparam() in the constructor if
	4	it's supported. This prevents breakage on CentOS7 which uses an old version of
	5	libmagic.
	6
	7	- Add tests for CentOS 7 & 8
	8
	9	Changes in 0.4.16 and 0.4.17
	10
	11	- add MAGIC_MIME_TYPE constant, use that in preference to MAGIC_MIME internally.
	12	This sets up for a breaking change in a future major version bump where
	13	MAGIC_MIME will change to mathch magic.h.
	14	- add magic.version() function to return library version
	15	- add setparam/getparam to control internal behavior
	16	- increase internal limits with setparam to prevent spurious error on some jpeg files
	17	- various setup.py improvements to declare modern python support
	18	- support MSYS2 magic dlls
	19	- fix warning about using 'is' on an int in python 3.8
	20	- include tests in source distribution
	21
	22	- many test improvements:
	23	-- tox runner support
	24	-- remove deprecated test_suite field from setup.py
	25	-- docker tests that cover all LTS ubuntu versions
	26	-- add test for snapp file identification
	27
	28	- doc improvements
	29	-- document dependency install process for debian
	30	-- various typos
	31	-- document test running process

+3

-2

MANIFEST.in less more

0	0	include *.py
1	1	include LICENSE
2		include test/testdata/*
3		include test/*.sh
	2	graft tests
	3	global-exclude __pycache__
	4	global-exclude *.py[co]

+60

-15

README.md less more

1	1	[![PyPI version](https://badge.fury.io/py/python-magic.svg)](https://badge.fury.io/py/python-magic)
2	2	[![Build Status](https://travis-ci.org/ahupp/python-magic.svg?branch=master)](https://travis-ci.org/ahupp/python-magic)
3	3
4		python-magic is a python interface to the libmagic file type
	4	python-magic is a Python interface to the libmagic file type
5	5	identification library. libmagic identifies file types by checking
6	6	their headers according to a predefined list of file types. This
7	7	functionality is exposed to the command line by the Unix command

13	13	>>> import magic
14	14	>>> magic.from_file("testdata/test.pdf")
15	15	'PDF document, version 1.2'
16		>>> magic.from_buffer(open("testdata/test.pdf").read(1024))
	16	# recommend using at least the first 2048 bytes, as less can produce incorrect identification
	17	>>> magic.from_buffer(open("testdata/test.pdf", "rb").read(2048))
17	18	'PDF document, version 1.2'
18	19	>>> magic.from_file("testdata/test.pdf", mime=True)
19	20	'application/pdf'

40	41	'text/plain'
41	42	```
42	43
43		## Name Conflict
44
45		There are, sadly, two libraries which use the module name `magic`. Both have been around for quite a while.If you are using this module and get an error using a method like `open`, your code is expecting the other one. Hopefully one day these will be reconciled.
46
47	44	## Installation
48	45
49		The current stable version of python-magic is available on pypi and
	46	The current stable version of python-magic is available on PyPI and
50	47	can be installed by running `pip install python-magic`.
51	48
52	49	Other sources:
53	50
54		- pypi: http://pypi.python.org/pypi/python-magic/
55		- github: https://github.com/ahupp/python-magic
	51	- PyPI: http://pypi.python.org/pypi/python-magic/
	52	- GitHub: https://github.com/ahupp/python-magic
56	53
57		### Dependencies
	54	This module is a simple wrapper around the libmagic C library, and
	55	that must be installed as well:
58	56
59		On Windows, copy magic1.dll, regex2.dll, and zlib1.dll onto your PATH from the Binaries and Dependencies zipfiles provided by the [File for Windows](http://gnuwin32.sourceforge.net/packages/file.htm) project. You will need to copy the file `magic` out of `[binary-zip]\share\misc`, and pass it's location to `Magic(magic_file=...)`. If you are using a 64-bit build of python, you'll need 64-bit libmagic binaries which can be found here: https://github.com/pidydx/libmagicwin64 (note: untested)
	57	### Debian/Ubuntu
60	58
61		On OSX:
	59	$ sudo apt-get install libmagic1
	60
	61	### Windows
	62
	63	You'll need DLLs for libmagic. @julian-r maintains a pypi package with the DLLs, you can fetch it with:
	64
	65	$ pip install python-magic-bin
	66
	67	### OSX
62	68
63	69	- When using Homebrew: `brew install libmagic`
64	70	- When using macports: `port install file`

72	78
73	79	- 'WindowsError: [Error 193] %1 is not a valid Win32 application':
74	80	Attempting to run the 32-bit libmagic DLL in a 64-bit build of
75		python will fail with this error. Here are 64-bit builds of libmagic for windows: https://github.com/pidydx/libmagicwin64
	81	python will fail with this error. Here are 64-bit builds of libmagic for windows: https://github.com/pidydx/libmagicwin64.
	82	Newer version can be found here: https://github.com/nscaife/file-windows.
76	83
77	84	- 'WindowsError: exception: access violation writing 0x00000000 ' This may indicate you are mixing
78	85	Windows Python and Cygwin Python. Make sure your libmagic and python builds are consistent.
	86
	87
	88	## Bug Reports
	89
	90	python-magic is a thin layer over the libmagic C library.
	91	Historically, most bugs that have been reported against python-magic
	92	are actually bugs in libmagic; libmagic bugs can be reported on their
	93	tracker here: https://bugs.astron.com/my_view_page.php. If you're not
	94	sure where the bug lies feel free to file an issue on GitHub and I can
	95	triage it.
	96
	97	## Running the tests
	98
	99	To run the tests across 3 recent Ubuntu LTS releases (depends on Docker):
	100
	101	$ ./test_docker.sh
	102
	103	To run tests locally across all available python versions:
	104
	105	$ ./test/run.py
	106
	107	To run against a specific python version:
	108
	109	$ LC_ALL=en_US.UTF-8 python3 test/test.py
	110
	111	## Versioning
	112
	113	Minor version bumps should be backwards compatible. Major bumps are not.
	114
	115	## Name Conflict
	116
	117	There are, sadly, two libraries which use the module name `magic`.
	118	Both have been around for quite a while. If you are using this module
	119	and get an error using a method like `open`, your code is expecting
	120	the other one. Hopefully one day these will be reconciled.
	121
79	122
80	123	## Author
81	124

84	127	switched to ctypes once that was part of the python standard library.
85	128
86	129	You can contact me via my [website](http://hupp.org/adam) or
87		[github](http://github.com/ahupp).
	130	[GitHub](http://github.com/ahupp).
88	131
89	132	## Contributors
90	133
91		Thanks to these folks on github who submitted features and bugfixes.
	134	Thanks to these folks on github who submitted features and bug fixes.
92	135
93	136	- Amit Sethi
94	137	- [bigben87](https://github.com/bigben87)
95	138	- [fallgesetz](https://github.com/fallgesetz)
96	139	- [FlaPer87](https://github.com/FlaPer87)
	140	- [Hugo van Kemenade](https://github.com/hugovk)
97	141	- [lukenowak](https://github.com/lukenowak)
98	142	- NicolasDelaby
99	143	- [email protected]

105	149	python-magic is distributed under the MIT license. See the included
106	150	LICENSE file for details.
107	151
	152	I am providing code in the repository to you under an open source license. Because this is my personal repository, the license you receive to my code is from me and not my employer (Facebook).
108	153

+154

-24

magic.py less more

13	13	'PDF document, version 1.2'
14	14	>>>
15	15
16
17	16	"""
18	17
19	18	import sys
20	19	import glob
21		import os.path
22	20	import ctypes
23	21	import ctypes.util
24	22	import threading
25	23
26		from ctypes import c_char_p, c_int, c_size_t, c_void_p
	24	from ctypes import c_char_p, c_int, c_size_t, c_void_p, byref, POINTER
27	25
28	26
29	27	class MagicException(Exception):

35	33	class Magic:
36	34	"""
37	35	Magic is a wrapper around the libmagic C library.
38
39	36	"""
40	37
41	38	def __init__(self, mime=False, magic_file=None, mime_encoding=False,
42		keep_going=False, uncompress=False):
	39	keep_going=False, uncompress=False, raw=False, extension=False):
43	40	"""
44	41	Create a new libmagic wrapper.
45	42

48	45	magic_file - use a mime database other than the system default
49	46	keep_going - don't stop at the first match, keep going
50	47	uncompress - Try to look inside compressed files.
	48	raw - Do not try to decode "non-printable" chars.
	49	extension - Print a slash-separated list of valid extensions for the file type found.
51	50	"""
	51
	52	self.cookie = None
52	53	self.flags = MAGIC_NONE
53	54	if mime:
54		self.flags \|= MAGIC_MIME
	55	self.flags \|= MAGIC_MIME_TYPE
55	56	if mime_encoding:
56	57	self.flags \|= MAGIC_MIME_ENCODING
57	58	if keep_going:
58	59	self.flags \|= MAGIC_CONTINUE
59
60	60	if uncompress:
61	61	self.flags \|= MAGIC_COMPRESS
	62	if raw:
	63	self.flags \|= MAGIC_RAW
	64	if extension:
	65	self.flags \|= MAGIC_EXTENSION
62	66
63	67	self.cookie = magic_open(self.flags)
64	68	self.lock = threading.Lock()
65
	69
66	70	magic_load(self.cookie, magic_file)
	71
	72	# MAGIC_EXTENSION was added in 523 or 524, so bail if
	73	# it doesn't appear to be available
	74	if extension and (not _has_version or version() < 524):
	75	raise NotImplementedError('MAGIC_EXTENSION is not supported in this version of libmagic')
	76
	77	# For https://github.com/ahupp/python-magic/issues/190
	78	# libmagic has fixed internal limits that some files exceed, causing
	79	# an error. We can avoid this (at least for the sample file given)
	80	# by bumping the limit up. It's not clear if this is a general solution
	81	# or whether other internal limits should be increased, but given
	82	# the lack of other reports I'll assume this is rare.
	83	if _has_param:
	84	try:
	85	self.setparam(MAGIC_PARAM_NAME_MAX, 64)
	86	except MagicException as e:
	87	# some versions of libmagic fail this call,
	88	# so rather than fail hard just use default behavior
	89	pass
67	90
68	91	def from_buffer(self, buf):
69	92	"""

71	94	"""
72	95	with self.lock:
73	96	try:
	97	# if we're on python3, convert buf to bytes
	98	# otherwise this string is passed as wchar*
	99	# which is not what libmagic expects
	100	if type(buf) == str and str != bytes:
	101	buf = buf.encode('utf-8', errors='replace')
74	102	return maybe_decode(magic_buffer(self.cookie, buf))
75	103	except MagicException as e:
76	104	return self._handle509Bug(e)

85	113	except MagicException as e:
86	114	return self._handle509Bug(e)
87	115
	116	def from_descriptor(self, fd):
	117	with self.lock:
	118	try:
	119	return maybe_decode(magic_descriptor(self.cookie, fd))
	120	except MagicException as e:
	121	return self._handle509Bug(e)
	122
88	123	def _handle509Bug(self, e):
89	124	# libmagic 5.09 has a bug where it might fail to identify the
90	125	# mimetype of a file and returns null from magic_file (and
91	126	# likely _buffer), but also does not return an error message.
92		if e.message is None and (self.flags & MAGIC_MIME):
	127	if e.message is None and (self.flags & MAGIC_MIME_TYPE):
93	128	return "application/octet-stream"
94	129	else:
95	130	raise e
96
	131
	132	def setparam(self, param, val):
	133	return magic_setparam(self.cookie, param, val)
	134
	135	def getparam(self, param):
	136	return magic_getparam(self.cookie, param)
	137
97	138	def __del__(self):
98	139	# no _thread_check here because there can be no other
99	140	# references to this object at this point.

111	152
112	153	_instances = {}
113	154
	155
114	156	def _get_magic_type(mime):
115	157	i = _instances.get(mime)
116	158	if i is None:
117	159	i = _instances[mime] = Magic(mime=mime)
118	160	return i
119	161
	162
120	163	def from_file(filename, mime=False):
121	164	""""
122	165	Accepts a filename and returns the detected filetype. Return

129	172	m = _get_magic_type(mime)
130	173	return m.from_file(filename)
131	174
	175
132	176	def from_buffer(buffer, mime=False):
133	177	"""
134	178	Accepts a binary string and returns the detected filetype. Return

142	186	return m.from_buffer(buffer)
143	187
144	188
	189	def from_descriptor(fd, mime=False):
	190	"""
	191	Accepts a file descriptor and returns the detected filetype. Return
	192	value is the mimetype if mime=True, otherwise a human readable
	193	name.
	194
	195	>>> f = open("testdata/test.pdf")
	196	>>> magic.from_descriptor(f.fileno())
	197	'PDF document, version 1.2'
	198	"""
	199	m = _get_magic_type(mime)
	200	return m.from_descriptor(fd)
145	201
146	202
147	203	libmagic = None
148	204	# Let's try to find magic or magic1
149		dll = ctypes.util.find_library('magic') or ctypes.util.find_library('magic1') or ctypes.util.find_library('cygmagic-1')
150
151		# This is necessary because find_library returns None if it doesn't find the library
	205	dll = ctypes.util.find_library('magic') \
	206	or ctypes.util.find_library('magic1') \
	207	or ctypes.util.find_library('cygmagic-1') \
	208	or ctypes.util.find_library('libmagic-1') \
	209	or ctypes.util.find_library('msys-magic-1') #for MSYS2
	210
	211	# necessary because find_library returns None if it doesn't find the library
152	212	if dll:
153	213	libmagic = ctypes.CDLL(dll)
154	214
155	215	if not libmagic or not libmagic._name:
156		windows_dlls = ['magic1.dll','cygmagic-1.dll']
	216	windows_dlls = ['magic1.dll', 'cygmagic-1.dll', 'libmagic-1.dll', 'msys-magic-1.dll']
157	217	platform_to_lib = {'darwin': ['/opt/local/lib/libmagic.dylib',
158	218	'/usr/local/lib/libmagic.dylib'] +
159		# Assumes there will only be one version installed
160		glob.glob('/usr/local/Cellar/libmagic/*/lib/libmagic.dylib'),
	219	# Assumes there will only be one version installed
	220	glob.glob('/usr/local/Cellar/libmagic/*/lib/libmagic.dylib'), # flake8:noqa
161	221	'win32': windows_dlls,
162	222	'cygwin': windows_dlls,
163		'linux': ['libmagic.so.1'], # fallback for some Linuxes (e.g. Alpine) where library search does not work
	223	'linux': ['libmagic.so.1'], # fallback for some Linuxes (e.g. Alpine) where library search does not work # flake8:noqa
164	224	}
165	225	platform = 'linux' if sys.platform.startswith('linux') else sys.platform
166	226	for dll in platform_to_lib.get(platform, []):

176	236
177	237	magic_t = ctypes.c_void_p
178	238
	239
179	240	def errorcheck_null(result, func, args):
180	241	if result is None:
181	242	err = magic_error(args[0])

183	244	else:
184	245	return result
185	246
	247
186	248	def errorcheck_negative_one(result, func, args):
187		if result is -1:
	249	if result == -1:
188	250	err = magic_error(args[0])
189	251	raise MagicException(err)
190	252	else:

197	259	if str == bytes:
198	260	return s
199	261	else:
200		return s.decode('utf-8')
201
	262	# backslashreplace here because sometimes libmagic will return metadata in the charset
	263	# of the file, which is unknown to us (e.g the title of a Word doc)
	264	return s.decode('utf-8', 'backslashreplace')
	265
	266
202	267	def coerce_filename(filename):
203	268	if filename is None:
204	269	return None
205	270
206	271	# ctypes will implicitly convert unicode strings to bytes with
207		# .encode('ascii'). If you use the filesystem encoding
	272	# .encode('ascii'). If you use the filesystem encoding
208	273	# then you'll get inconsistent behavior (crashes) depending on the user's
209	274	# LANG environment variable
210	275	is_unicode = (sys.version_info[0] <= 2 and

212	277	(sys.version_info[0] >= 3 and
213	278	isinstance(filename, str))
214	279	if is_unicode:
215		return filename.encode('utf-8')
	280	return filename.encode('utf-8', 'surrogateescape')
216	281	else:
217	282	return filename
	283
218	284
219	285	magic_open = libmagic.magic_open
220	286	magic_open.restype = magic_t

237	303	_magic_file.argtypes = [magic_t, c_char_p]
238	304	_magic_file.errcheck = errorcheck_null
239	305
	306
240	307	def magic_file(cookie, filename):
241	308	return _magic_file(cookie, coerce_filename(filename))
	309
242	310
243	311	_magic_buffer = libmagic.magic_buffer
244	312	_magic_buffer.restype = c_char_p
245	313	_magic_buffer.argtypes = [magic_t, c_void_p, c_size_t]
246	314	_magic_buffer.errcheck = errorcheck_null
247	315
	316
248	317	def magic_buffer(cookie, buf):
249	318	return _magic_buffer(cookie, buf, len(buf))
	319
	320
	321	_magic_descriptor = libmagic.magic_descriptor
	322	_magic_descriptor.restype = c_char_p
	323	_magic_descriptor.argtypes = [magic_t, c_int]
	324	_magic_descriptor.errcheck = errorcheck_null
	325
	326
	327	def magic_descriptor(cookie, fd):
	328	return _magic_descriptor(cookie, fd)
250	329
251	330
252	331	_magic_load = libmagic.magic_load

254	333	_magic_load.argtypes = [magic_t, c_char_p]
255	334	_magic_load.errcheck = errorcheck_negative_one
256	335
	336
257	337	def magic_load(cookie, filename):
258	338	return _magic_load(cookie, coerce_filename(filename))
	339
259	340
260	341	magic_setflags = libmagic.magic_setflags
261	342	magic_setflags.restype = c_int

269	350	magic_compile.restype = c_int
270	351	magic_compile.argtypes = [magic_t, c_char_p]
271	352
272
	353	_has_param = False
	354	if hasattr(libmagic, 'magic_setparam') and hasattr(libmagic, 'magic_getparam'):
	355	_has_param = True
	356	_magic_setparam = libmagic.magic_setparam
	357	_magic_setparam.restype = c_int
	358	_magic_setparam.argtypes = [magic_t, c_int, POINTER(c_size_t)]
	359	_magic_setparam.errcheck = errorcheck_negative_one
	360
	361	_magic_getparam = libmagic.magic_getparam
	362	_magic_getparam.restype = c_int
	363	_magic_getparam.argtypes = [magic_t, c_int, POINTER(c_size_t)]
	364	_magic_getparam.errcheck = errorcheck_negative_one
	365
	366	def magic_setparam(cookie, param, val):
	367	if not _has_param:
	368	raise NotImplementedError("magic_setparam not implemented")
	369	v = c_size_t(val)
	370	return _magic_setparam(cookie, param, byref(v))
	371
	372	def magic_getparam(cookie, param):
	373	if not _has_param:
	374	raise NotImplementedError("magic_getparam not implemented")
	375	val = c_size_t()
	376	_magic_getparam(cookie, param, byref(val))
	377	return val.value
	378
	379	_has_version = False
	380	if hasattr(libmagic, "magic_version"):
	381	_has_version = True
	382	magic_version = libmagic.magic_version
	383	magic_version.restype = c_int
	384	magic_version.argtypes = []
	385
	386	def version():
	387	if not _has_version:
	388	raise NotImplementedError("magic_version not implemented")
	389	return magic_version()
273	390
274	391	MAGIC_NONE = 0x000000 # No flags
275	392	MAGIC_DEBUG = 0x000001 # Turn on debugging
276	393	MAGIC_SYMLINK = 0x000002 # Follow symlinks
277	394	MAGIC_COMPRESS = 0x000004 # Check inside compressed files
278	395	MAGIC_DEVICES = 0x000008 # Look at the contents of devices
	396	MAGIC_MIME_TYPE = 0x000010 # Return a mime string
	397	MAGIC_MIME_ENCODING = 0x000400 # Return the MIME encoding
	398	# TODO: should be
	399	# MAGIC_MIME = MAGIC_MIME_TYPE \| MAGIC_MIME_ENCODING
279	400	MAGIC_MIME = 0x000010 # Return a mime string
280		MAGIC_MIME_ENCODING = 0x000400 # Return the MIME encoding
	401	MAGIC_EXTENSION = 0x1000000 # Return a /-separated list of extensions
	402
281	403	MAGIC_CONTINUE = 0x000020 # Return all matches
282	404	MAGIC_CHECK = 0x000040 # Print warnings to stderr
283	405	MAGIC_PRESERVE_ATIME = 0x000080 # Restore access time on exit

293	415	MAGIC_NO_CHECK_TROFF = 0x040000 # Don't check ascii/troff
294	416	MAGIC_NO_CHECK_FORTRAN = 0x080000 # Don't check ascii/fortran
295	417	MAGIC_NO_CHECK_TOKENS = 0x100000 # Don't check ascii/tokens
	418
	419	MAGIC_PARAM_INDIR_MAX = 0 # Recursion limit for indirect magic
	420	MAGIC_PARAM_NAME_MAX = 1 # Use count limit for name/use magic
	421	MAGIC_PARAM_ELF_PHNUM_MAX = 2 # Max ELF notes processed
	422	MAGIC_PARAM_ELF_SHNUM_MAX = 3 # Max ELF program sections processed
	423	MAGIC_PARAM_ELF_NOTES_MAX = 4 # # Max ELF sections processed
	424	MAGIC_PARAM_REGEX_MAX = 5 # Length limit for regex searches
	425	MAGIC_PARAM_BYTES_MAX = 6 # Max number of bytes to read from file

+86

-0

magic.pyi less more

	0	import ctypes.util
	1	import threading
	2	from typing import Any, Text, Optional, Union
	3
	4	class MagicException(Exception):
	5	message: Any = ...
	6	def __init__(self, message: Any) -> None: ...
	7
	8	class Magic:
	9	flags: int = ...
	10	cookie: Any = ...
	11	lock: threading.Lock = ...
	12	def __init__(self, mime: bool = ..., magic_file: Optional[Any] = ..., mime_encoding: bool = ..., keep_going: bool = ..., uncompress: bool = ..., raw: bool = ...) -> None: ...
	13	def from_buffer(self, buf: Union[bytes, str]) -> Text: ...
	14	def from_file(self, filename: Union[bytes, str]) -> Text: ...
	15	def from_descriptor(self, fd: int, mime: bool = ...) -> Text: ...
	16	def setparam(self, param: Any, val: Any): ...
	17	def getparam(self, param: Any): ...
	18	def __del__(self) -> None: ...
	19
	20	def from_file(filename: Union[bytes, str], mime: bool = ...) -> Text: ...
	21	def from_buffer(buffer: Union[bytes, str], mime: bool = ...) -> Text: ...
	22	def from_descriptor(fd: int, mime: bool = ...) -> Text: ...
	23
	24	libmagic: Any
	25	dll: Any
	26	windows_dlls: Any
	27	platform_to_lib: Any
	28	platform: Any
	29	magic_t = ctypes.c_void_p
	30
	31	def errorcheck_null(result: Any, func: Any, args: Any): ...
	32	def errorcheck_negative_one(result: Any, func: Any, args: Any): ...
	33	def maybe_decode(s: Union[bytes, str]) -> str: ...
	34	def coerce_filename(filename: Any): ...
	35
	36	magic_open: Any
	37	magic_close: Any
	38	magic_error: Any
	39	magic_errno: Any
	40
	41	def magic_file(cookie: Any, filename: Any): ...
	42	def magic_buffer(cookie: Any, buf: Any): ...
	43	def magic_descriptor(cookie: Any, fd: int): ...
	44	def magic_load(cookie: Any, filename: Any): ...
	45
	46	magic_setflags: Any
	47	magic_check: Any
	48	magic_compile: Any
	49
	50	def magic_setparam(cookie: Any, param: Any, val: Any): ...
	51	def magic_getparam(cookie: Any, param: Any): ...
	52
	53	magic_version: Any
	54
	55	def version(): ...
	56
	57	MAGIC_NONE: int
	58	MAGIC_DEBUG: int
	59	MAGIC_SYMLINK: int
	60	MAGIC_COMPRESS: int
	61	MAGIC_DEVICES: int
	62	MAGIC_MIME_TYPE: int
	63	MAGIC_MIME_ENCODING: int
	64	MAGIC_MIME: int
	65	MAGIC_CONTINUE: int
	66	MAGIC_CHECK: int
	67	MAGIC_PRESERVE_ATIME: int
	68	MAGIC_RAW: int
	69	MAGIC_ERROR: int
	70	MAGIC_NO_CHECK_COMPRESS: int
	71	MAGIC_NO_CHECK_TAR: int
	72	MAGIC_NO_CHECK_SOFT: int
	73	MAGIC_NO_CHECK_APPTYPE: int
	74	MAGIC_NO_CHECK_ELF: int
	75	MAGIC_NO_CHECK_ASCII: int
	76	MAGIC_NO_CHECK_TROFF: int
	77	MAGIC_NO_CHECK_FORTRAN: int
	78	MAGIC_NO_CHECK_TOKENS: int
	79	MAGIC_PARAM_INDIR_MAX: int
	80	MAGIC_PARAM_NAME_MAX: int
	81	MAGIC_PARAM_ELF_PHNUM_MAX: int
	82	MAGIC_PARAM_ELF_SHNUM_MAX: int
	83	MAGIC_PARAM_ELF_NOTES_MAX: int
	84	MAGIC_PARAM_REGEX_MAX: int
	85	MAGIC_PARAM_BYTES_MAX: int

+19

-6

setup.py less more

1	1	# -- coding: utf-8 --
2	2
3	3	from setuptools import setup
	4	import io
	5	import os
	6
	7
	8	def read(file_name):
	9	"""Read a text file and return the content as a string."""
	10	with io.open(os.path.join(os.path.dirname(__file__), file_name),
	11	encoding='utf-8') as f:
	12	return f.read()
4	13
5	14	setup(name='python-magic',
6	15	description='File type identification using libmagic',
7	16	author='Adam Hupp',
8	17	author_email='[email protected]',
9	18	url="http://github.com/ahupp/python-magic",
10		version='0.4.13',
	19	version='0.4.18',
11	20	py_modules=['magic'],
12		long_description="""This module uses ctypes to access the libmagic file type
13		identification library. It makes use of the local magic database and
14		supports both textual and MIME-type output.
15		""",
	21	long_description=read('README.md'),
	22	long_description_content_type='text/markdown',
16	23	keywords="mime magic file",
17	24	license="MIT",
18		test_suite='test',
	25	python_requires='>=2.7, !=3.0., !=3.1., !=3.2., !=3.3., !=3.4.*',
19	26	classifiers=[
20	27	'Intended Audience :: Developers',
21	28	'License :: OSI Approved :: MIT License',
22	29	'Programming Language :: Python',
23	30	'Programming Language :: Python :: 2',
	31	'Programming Language :: Python :: 2.7',
24	32	'Programming Language :: Python :: 3',
	33	'Programming Language :: Python :: 3.5',
	34	'Programming Language :: Python :: 3.6',
	35	'Programming Language :: Python :: 3.7',
	36	'Programming Language :: Python :: 3.8',
	37	'Programming Language :: Python :: Implementation :: CPython',
25	38	],
26	39	)

+5

-0

test/Dockerfile_archlinux less more

	0	FROM archlinux:20200505
	1	RUN yes \| pacman -Syyu --overwrite '*'
	2	RUN yes \| pacman -S python python2 file which
	3	COPY . /python-magic
	4	CMD cd /python-magic/test && python3 ./run.py

+8

-0

test/Dockerfile_bionic less more

	0	FROM ubuntu:bionic
	1	RUN apt-get update
	2	RUN apt-get -y install python
	3	RUN apt-get -y install python3
	4	RUN apt-get -y install locales
	5	RUN locale-gen en_US.UTF-8
	6	COPY . /python-magic
	7	CMD cd /python-magic/test && python3 ./run.py

+5

-0

test/Dockerfile_centos7 less more

	0	FROM centos:7
	1	RUN yum -y update
	2	RUN yum -y install file-devel python3 python2 which
	3	COPY . /python-magic
	4	CMD cd /python-magic/test && python3 ./run.py

+5

-0

test/Dockerfile_centos8 less more

	0	FROM centos:8
	1	RUN yum -y update
	2	RUN yum -y install file-libs python3 python2 which
	3	COPY . /python-magic
	4	CMD cd /python-magic/test && python3 ./run.py

+8

-0

test/Dockerfile_focal less more

	0	FROM ubuntu:focal
	1	RUN apt-get update
	2	RUN apt-get -y install python2
	3	RUN apt-get -y install python3
	4	RUN apt-get -y install locales
	5	RUN locale-gen en_US.UTF-8
	6	COPY . /python-magic
	7	CMD cd /python-magic/test && python3 ./run.py

+8

-0

test/Dockerfile_xenial less more

	0	FROM ubuntu:xenial
	1	RUN apt-get update
	2	RUN apt-get -y install python
	3	RUN apt-get -y install python3
	4	RUN apt-get -y install locales
	5	RUN locale-gen en_US.UTF-8
	6	COPY . /python-magic
	7	CMD cd /python-magic/test && python3 ./run.py

+10

-0

test/README less more

	0	To run the tests across a selection of Ubuntu LTS versions:
	1
	2	docker build -t "python_magic/xenial:latest" -f test/Dockerfile_xenial .
	3	docker build -t "python_magic/bionic:latest" -f test/Dockerfile_bionic .
	4	docker build -t "python_magic/focal:latest" -f test/Dockerfile_focal .
	5
	6	docker run python_magic/xenial:latest
	7	docker run python_magic/bionic:latest
	8	docker run python_magic/focal:latest
	9

+32

-0

test/run.py less more

	0
	1	import subprocess
	2	import os.path
	3	import sys
	4
	5	this_dir = os.path.dirname(sys.argv[0])
	6
	7	new_env = {
	8	'LC_ALL': 'en_US.UTF-8',
	9	'PYTHONPATH': os.path.join(this_dir, ".."),
	10	}
	11
	12	def has_py(version):
	13	ret = subprocess.run("which %s" % version, shell=True, stdout=subprocess.DEVNULL)
	14	return ret.returncode == 0
	15
	16	def run_test(versions):
	17
	18	found = False
	19	for i in versions:
	20	if not has_py(i):
	21	# if this version doesn't exist in path, skip
	22	continue
	23	found = True
	24	print("Testing %s" % i)
	25	subprocess.run([i, os.path.join(this_dir, "test.py")], env=new_env, check=True)
	26
	27	if not found:
	28	sys.exit("No versions found: " + str(versions))
	29
	30	run_test(["python2", "python2.7"])
	31	run_test(["python3.5", "python3.6", "python3.7", "python3.8"])

+0

-12

~~test/run.sh~~ less more

0		#!/bin/sh
1
2		set -e
3
4		# ensure we can use unicode filenames in the test
5		export LC_ALL=en_US.UTF-8
6		THISDIR=`dirname $0`
7		export PYTHONPATH=${THISDIR}/..
8
9		python2.6 ${THISDIR}/test.py
10		python2.7 ${THISDIR}/test.py
11		python3 ${THISDIR}/test.py

+141

-33

test/test.py less more

0		import os, sys
	0	import os
1	1	# for output which reports a local time
2	2	os.environ['TZ'] = 'GMT'
	3
	4	if os.environ.get('LC_ALL','') != 'en_US.UTF-8':
	5	# this ensure we're in a utf-8 default filesystem encoding which is
	6	# necessary for some tests
	7	raise Exception("must run `export LC_ALL=en_US.UTF-8` before running test suite")
	8
3	9	import shutil
4	10	import os.path
5	11	import unittest
6	12
7	13	import magic
	14	import sys
8	15
9	16	class MagicTest(unittest.TestCase):
10		TESTDATA_DIR = os.path.join(os.path.dirname(__file__), 'testdata')
11
12		def assert_values(self, m, expected_values):
	17	TESTDATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'testdata')
	18
	19	def test_version(self):
	20	try:
	21	self.assertTrue(magic.version() > 0)
	22	except NotImplementedError:
	23	pass
	24
	25	def test_fs_encoding(self):
	26	self.assertEqual('utf-8', sys.getfilesystemencoding().lower())
	27
	28	def assert_values(self, m, expected_values, buf_equals_file=True):
13	29	for filename, expected_value in expected_values.items():
14	30	try:
15	31	filename = os.path.join(self.TESTDATA_DIR, filename)
16	32	except TypeError:
17		filename = os.path.join(self.TESTDATA_DIR.encode('utf-8'), filename)
18
19
	33	filename = os.path.join(
	34	self.TESTDATA_DIR.encode('utf-8'), filename)
	35
20	36	if type(expected_value) is not tuple:
21	37	expected_value = (expected_value,)
22	38
23		for i in expected_value:
24		with open(filename, 'rb') as f:
25		buf_value = m.from_buffer(f.read())
26
27		file_value = m.from_file(filename)
28		if buf_value == i and file_value == i:
29		break
30		else:
31		self.assertTrue(False, "no match for " + repr(expected_value))
32
	39	with open(filename, 'rb') as f:
	40	buf_value = m.from_buffer(f.read())
	41
	42	file_value = m.from_file(filename)
	43
	44	if buf_equals_file:
	45	self.assertEqual(buf_value, file_value)
	46
	47	for value in (buf_value, file_value):
	48	self.assertIn(value, expected_value)
	49
	50	def test_from_file_str_and_bytes(self):
	51	filename = os.path.join(self.TESTDATA_DIR, "test.pdf")
	52
	53	self.assertEqual('application/pdf',
	54	magic.from_file(filename, mime=True))
	55	self.assertEqual('application/pdf',
	56	magic.from_file(filename.encode('utf-8'), mime=True))
	57
	58	def test_from_descriptor_str_and_bytes(self):
	59	with open(os.path.join(self.TESTDATA_DIR, "test.pdf")) as f:
	60	self.assertEqual('application/pdf',
	61	magic.from_descriptor(f.fileno(), mime=True))
	62	self.assertEqual('application/pdf',
	63	magic.from_descriptor(f.fileno(), mime=True))
	64
	65	def test_from_buffer_str_and_bytes(self):
	66	m = magic.Magic(mime=True)
	67
	68	self.assertTrue(
	69	m.from_buffer('#!/usr/bin/env python\nprint("foo")')
	70	in ("text/x-python", "text/x-script.python"))
	71	self.assertTrue(
	72	m.from_buffer(b'#!/usr/bin/env python\nprint("foo")')
	73	in ("text/x-python", "text/x-script.python"))
	74
	75
33	76	def test_mime_types(self):
34		dest = os.path.join(MagicTest.TESTDATA_DIR, b'\xce\xbb'.decode('utf-8'))
	77	dest = os.path.join(MagicTest.TESTDATA_DIR,
	78	b'\xce\xbb'.decode('utf-8'))
35	79	shutil.copyfile(os.path.join(MagicTest.TESTDATA_DIR, 'lambda'), dest)
36	80	try:
37	81	m = magic.Magic(mime=True)
38	82	self.assert_values(m, {
39		'magic.pyc': 'application/octet-stream',
	83	'magic._pyc_': ('application/octet-stream', 'text/x-bytecode.python'),
40	84	'test.pdf': 'application/pdf',
41		'test.gz': 'application/gzip',
	85	'test.gz': ('application/gzip', 'application/x-gzip'),
	86	'test.snappy.parquet': 'application/octet-stream',
42	87	'text.txt': 'text/plain',
43	88	b'\xce\xbb'.decode('utf-8'): 'text/plain',
44	89	b'\xce\xbb': 'text/plain',

48	93
49	94	def test_descriptions(self):
50	95	m = magic.Magic()
51		os.environ['TZ'] = 'UTC' # To get the last modified date of test.gz in UTC
	96	os.environ['TZ'] = 'UTC' # To get last modified date of test.gz in UTC
52	97	try:
53	98	self.assert_values(m, {
54		'magic.pyc': 'python 2.4 byte-compiled',
	99	'magic._pyc_': 'python 2.4 byte-compiled',
55	100	'test.pdf': 'PDF document, version 1.2',
56	101	'test.gz':
57		('gzip compressed data, was "test", from Unix, last modified: Sun Jun 29 01:32:52 2008',
58		'gzip compressed data, was "test", last modified: Sun Jun 29 01:32:52 2008, from Unix'),
	102	('gzip compressed data, was "test", from Unix, last '
	103	'modified: Sun Jun 29 01:32:52 2008',
	104	'gzip compressed data, was "test", last modified'
	105	': Sun Jun 29 01:32:52 2008, from Unix',
	106	'gzip compressed data, was "test", last modified'
	107	': Sun Jun 29 01:32:52 2008, from Unix, original size 15',
	108	'gzip compressed data, was "test", '
	109	'last modified: Sun Jun 29 01:32:52 2008, '
	110	'from Unix, original size modulo 2^32 15',
	111	'gzip compressed data, was "test", last modified'
	112	': Sun Jun 29 01:32:52 2008, from Unix, truncated'
	113	),
59	114	'text.txt': 'ASCII text',
	115	'test.snappy.parquet': ('Apache Parquet', 'Par archive data'),
	116	}, buf_equals_file=False)
	117	finally:
	118	del os.environ['TZ']
	119
	120	def test_extension(self):
	121	try:
	122	m = magic.Magic(extension=True)
	123	self.assert_values(m, {
	124	# some versions return '' for the extensions of a gz file,
	125	# including w/ the command line. Who knows...
	126	'test.gz': ('gz/tgz/tpz/zabw/svgz', '', '???'),
	127	'name_use.jpg': 'jpeg/jpg/jpe/jfif',
60	128	})
61		finally:
62		del os.environ['TZ']
	129	except NotImplementedError:
	130	self.skipTest('MAGIC_EXTENSION not supported in this version')
	131
	132	def test_unicode_result_nonraw(self):
	133	m = magic.Magic(raw=False)
	134	src = os.path.join(MagicTest.TESTDATA_DIR, 'pgpunicode')
	135	result = m.from_file(src)
	136	# NOTE: This check is added as otherwise some magic files don't identify the test case as a PGP key.
	137	if 'PGP' in result:
	138	assert r"PGP\011Secret Sub-key -" == result
	139	else:
	140	raise unittest.SkipTest("Magic file doesn't return expected type.")
	141
	142	def test_unicode_result_raw(self):
	143	m = magic.Magic(raw=True)
	144	src = os.path.join(MagicTest.TESTDATA_DIR, 'pgpunicode')
	145	result = m.from_file(src)
	146	if 'PGP' in result:
	147	assert b'PGP\tSecret Sub-key -' == result.encode('utf-8')
	148	else:
	149	raise unittest.SkipTest("Magic file doesn't return expected type.")
63	150
64	151	def test_mime_encodings(self):
65	152	m = magic.Magic(mime_encoding=True)

84	171
85	172	m = magic.Magic(mime=True)
86	173	self.assertEqual(m.from_file(filename), 'image/jpeg')
87
88		m = magic.Magic(mime=True, keep_going=True)
89		self.assertEqual(m.from_file(filename), 'image/jpeg')
90
	174
	175	try:
	176	# this will throw if you have an "old" version of the library
	177	# I'm otherwise not sure how to query if keep_going is supported
	178	magic.version()
	179	m = magic.Magic(mime=True, keep_going=True)
	180	self.assertEqual(m.from_file(filename),
	181	'image/jpeg\\012- application/octet-stream')
	182	except NotImplementedError:
	183	pass
91	184
92	185	def test_rethrow(self):
93	186	old = magic.magic_buffer
94	187	try:
95		def t(x,y):
	188	def t(x, y):
96	189	raise magic.MagicException("passthrough")
97	190	magic.magic_buffer = t
98
99		self.assertRaises(magic.MagicException, magic.from_buffer, "hello", True)
	191
	192	with self.assertRaises(magic.MagicException):
	193	magic.from_buffer("hello", True)
100	194	finally:
101	195	magic.magic_buffer = old
	196
	197	def test_getparam(self):
	198	m = magic.Magic(mime=True)
	199	try:
	200	m.setparam(magic.MAGIC_PARAM_INDIR_MAX, 1)
	201	self.assertEqual(m.getparam(magic.MAGIC_PARAM_INDIR_MAX), 1)
	202	except NotImplementedError:
	203	pass
	204
	205	def test_name_count(self):
	206	m = magic.Magic()
	207	with open(os.path.join(self.TESTDATA_DIR, 'name_use.jpg'), 'rb') as f:
	208	m.from_buffer(f.read())
	209
102	210	if __name__ == '__main__':
103	211	unittest.main()

test/testdata/keep-going.jpg less more

Binary diff not shown

test/testdata/magic._pyc_ less more

Binary diff not shown

~~test/testdata/magic.pyc~~ less more

Binary diff not shown

test/testdata/name_use.jpg less more

Binary diff not shown

+1

-0

test/testdata/pgpunicode less more

0

qÊ⏎

test/testdata/test.snappy.parquet less more

Binary diff not shown

+11

-0

test.ps1 less more

	0
	1
	2	function TestInContainer($name) {
	3	$TAG="python_magic/${name}:latest"
	4	docker build -t $TAG -f "test/Dockerfile_${name}" .
	5	docker run "python_magic/${name}:latest"
	6	}
	7
	8	TestInContainer "xenial"
	9	TestInContainer "bionic"
	10	TestInContainer "focal"⏎

+21

-0

test_docker.sh less more

	0	#!/bin/bash
	1
	2	# Test with various versions of ubuntu. This more or less re-creates the
	3	# Travis CI test environment
	4
	5	set -e
	6
	7	function TestInContainer {
	8	local name="$1"
	9	local TAG="python_magic/${name}:latest"
	10	docker build -t $TAG -f "test/Dockerfile_${name}" .
	11	docker run "python_magic/${name}:latest"
	12	}
	13
	14	TestInContainer "xenial"
	15	TestInContainer "bionic"
	16	TestInContainer "focal"
	17	TestInContainer "centos7"
	18	TestInContainer "centos8"
	19	TestInContainer "archlinux"
	20

+46

-0

tox.ini less more

	0	[tox]
	1	envlist =
	2	coverage-clean,
	3	py27,
	4	py35,
	5	py36,
	6	py37,
	7	py38,
	8	coverage-report,
	9	mypy
	10
	11	[testenv]
	12	commands =
	13	coverage run --source=magic ./test/test.py
	14
	15	setenv =
	16	COVERAGE_FILE=.coverage.{envname}
	17	LC_ALL=en_US.UTF-8
	18	deps =
	19	.[test]
	20	zope.testrunner
	21	coverage
	22
	23	[testenv:coverage-clean]
	24	deps = coverage
	25	setenv =
	26	COVERAGE_FILE=.coverage
	27	skip_install = true
	28	commands = coverage erase
	29
	30	[testenv:coverage-report]
	31	deps = coverage
	32	setenv =
	33	COVERAGE_FILE=.coverage
	34	skip_install = true
	35	commands =
	36	coverage combine
	37	coverage report
	38	coverage html
	39	coverage
	40
	41	[testenv:mypy]
	42	deps = mypy
	43	skip_install = true
	44	commands =
	45	mypy magic.pyi

+3

-0

upload.sh less more

	0	#!/bin/sh
	1
	2	python setup.py sdist bdist_wheel upload