来源：大数据应用
本文约5700字，建议阅读11分钟
本文我们将深入探讨每个数据科学家都应该了解的12个Python特性。
Image Source: carbonAs a data scientist, you are no stranger to the powerful capabilities of Python. From data wrangling to machine learning, Python has become the de facto language of data science. But are you leveraging all the features that Python has to offer?In this article, we will dive deep into the 12 Python features that every data scientist should know. From comprehensions to data classes, these features will help you write more efficient, readable, and maintainable code.01 ComprehensionsComprehensions in Python are useful tools for machine learning and data science tasks as they allow you to create complex data structures in a concise and readable manner.List comprehensions can be used to generate lists of data, such as creating a list of squared values from a range of numbers. Nested list comprehensions can be used to flatten multi-dimensional arrays, a common preprocessing task in data science.
# list comprehension
_list = [x**2 for x in range(1, 11)]
# nested list comprehension to flatten list
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat_list = [num # append to list
for row in matrix # outer loop
for num in row] # inner loop
print(_list)
print(flat_list)Dictionary and set comprehensions are used to create dictionaries and sets, respectively. For example, a dictionary comprehension can be used to create a dictionary of feature names and their corresponding values in a machine learning model.Generator comprehensions are particularly useful for handling large datasets as they generate values on the fly rather than creating large data structures in memory. This helps improve performance and reduce memory usage.
# dictionary comprehension
_dict = {var: var ** 2 for var in range(1, 11) if var % 2 != 0}
# set comprehension
# create a set of squares of numbers from 1 to 10
_set = {x**2 for x in range(1, 11)}
# generator comprehension
_gen = (x**2 for x in range(1, 11))
print(_dict)
print(_set)
print(list(g for g in _gen))02 EnumerateEnumerate is a built-in function that allows you to iterate over a sequence (like a list or tuple) while keeping track of the index of each element.This is very useful when working with datasets as it allows easy access and manipulation of individual elements while keeping track of their index positions.Here we use enumerate to iterate over a list of strings and print the value if the index is even.
for idx, value in enumerate(["a", "b", "c", "d"]):
if idx % 2 == 0:
print(value)03 ZipZip is a built-in function that allows you to iterate over multiple sequences (like lists or tuples) in parallel.Below, we use zip to iterate over two lists x and y simultaneously and perform operations on their corresponding elements.
x = [1, 2, 3, 4]
y = [5, 6, 7, 8]
# iterate over both arrays simultaneously
for a, b in zip(x, y):
print(a, b, a + b, a * b)In this example, it prints the values of each element in x and y, their sum, and their product.04 GeneratorsGenerators in Python are a type of iterable that allows you to dynamically generate a sequence of values instead of generating all values at once and storing them in memory.This makes them very useful for handling large datasets that cannot fit into memory, as data is processed in small chunks or batches rather than all at once.Below, we use a generator function to generate the first n numbers in the Fibonacci sequence. The yield keyword is used to generate each value in the sequence one at a time instead of generating the entire sequence at once.
def fib_gen(n):
a, b = 0, 1
for _ in range(n):
yield a
a, b = b, a + b
res = fib_gen(10)
print(list(r for r in res))05 Lambda FunctionsLambda is a keyword used to create anonymous functions, which are functions without a name and can be defined in a single line of code.They are useful for dynamically defining custom functions for feature engineering, data preprocessing, or model evaluation.Below we create a simple function using lambda to filter even numbers from a list of numbers.
numbers = range(10)
even_numbers = list(filter(lambda x: x % 2 == 0, numbers))
print(even_numbers)This is another code snippet using lambda functions in Pandas.
import pandas as pd
data = { "sales_person": ["Alice", "Bob", "Charlie", "David"], "sale_amount": [100, 200, 300, 400],}
df = pd.DataFrame(data)
threshold = 250
df["above_threshold"] = df["sale_amount"].apply( lambda x: True if x >= threshold else False)
df06 Map, Filter, ReduceThe functions map, filter, and reduce are three built-in functions used for operating on and transforming data.Map is used to apply a function to each element of an iterable, filter is used to select elements from an iterable based on a condition, and reduce is used to apply a function to pairs of elements in an iterable to produce a single result.Below is an example that combines all three functions to compute the sum of squares of even numbers.
numbers = range(10)
# Use map(), filter(), and reduce() to preprocess and aggregate the list of numbers
even_numbers = filter(lambda x: x % 2 == 0, numbers)
squares = map(lambda x: x**2, even_numbers)
sum_of_squares = reduce(lambda x, y: x + y, squares)
print(f"Sum of the squares of even numbers: {sum_of_squares}")07 Any and AllAny and all are built-in functions that allow you to check if any or all elements in an iterable satisfy a specific condition.Any and all can be used to check if there are any missing values in a dataset or if all values in a column are within a certain range.Below is a simple example that checks for the presence of even values and all odd values.
data = [1, 3, 5, 7]
print(any(x % 2 == 0 for x in data))
print(all(x % 2 == 1 for x in data))08 NextNext is used to retrieve the next item from an iterator. An iterator is an object that can be iterated (looped) over, such as a list, tuple, set, or dictionary.Next is commonly used in data science to iterate over iterator or generator objects. It allows users to retrieve the next item from an iterable and is very useful for processing large datasets or streaming data.Below, we define a generator random_numbers() that generates random numbers between 0 and 1. Then we use the next() function to find the first number greater than 0.9 in the generator.
import random
def random_numbers():
while True:
yield random.random()
# Use next() to find the first number greater than 0.9
num = next(x for x in random_numbers() if x > 0.9)
print(f"First number greater than 0.9: {num}")09 Default Dictionary
Defaultdict is a subclass of the built-in dict class that allows you to provide default values for missing keys.
Defaultdict is very useful for handling missing or incomplete data, such as when working with sparse matrices or feature vectors. It can also be used to count frequencies of categorical variables.
An example is counting the occurrences of items in a list. If the parameter passed to default_factory is int, the value corresponding to the key is initialized to 0.
from collections import defaultdict
count = defaultdict(int)
for item in ['a', 'b', 'a', 'c', 'b', 'a']:
count[item] += 1
count10 PartialPartial is a function in the functools module that allows you to create a new function from an existing function with certain parameters pre-filled.Partial is very useful for creating custom functions or data transformations with specific parameters or arguments pre-filled. This helps reduce the amount of boilerplate code needed when defining and calling functions.Here, we use partial to create a new function increment from an existing function add, fixing one of its parameters to the value 1.Calling increment(1) essentially calls add(1, 1).
from functools import partial
def add(x, y):
return x + y
increment = partial(add, 1)
increment(1)11 LRU Cache
LRU cache is a decorator function in the functools module that allows caching the results of a function with a limited size cache.
LRU cache is very useful for optimizing functions that are computationally expensive or that may be called multiple times with the same parameters during model training processes.
Caching can help speed up function execution and reduce overall computational costs.
Here’s an example of using caching to efficiently compute Fibonacci numbers (https://en.wikipedia.org/wiki/Fibonacci_number) (also known as memoization in computer science).
from functools import lru_cache
@lru_cache(maxsize=None)
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
fibonacci(1e3)12 Data ClassesThe @dataclass decorator automatically generates several special methods for classes based on defined attributes, such as __init__, __repr__, and __eq__.This helps reduce the amount of boilerplate code needed when defining classes. Dataclass objects can represent data points, feature vectors, or model parameters, etc.In this example, a dataclass is used to define a simple class Person with three attributes: name, age, and city.
from dataclasses import dataclass
@dataclass
class Person:
name: str
age: int
city: str
p = Person("Alice", 30, "New York")
print(p)Original link:https://medium.com/bitgrit-data-science-publication/12-python-features-every-data-scientist-should-know-1d233dbaab0cEditor: Wang Jing
