Comprehensive Summary of Key Python Knowledge

This is a summary of key Python knowledge compiled by developer @Twenty-One on SegmentFault. Since it summarizes a lot of information, the length is a bit long. This is also something the author has ‘patched together’ for a long time, and I strongly recommend saving it to read slowly~

【Tutorial The way to receive is at the end!!】

Py2 VS Py3

Differences Between Py2 and Py3

• print became a function; in Python 2 it is a keyword.
• No more unicode objects; the default str is unicode.
• Division in Python 3 returns a float.
• There is no long type.
• xrange does not exist; range replaces xrange.
• You can use Chinese to define function names and variable names.
• Advanced unpacking and * unpacking.
• Keyword arguments after * must be given as name=value.
• raise from.
• iteritems removed, replaced by items().
• yield from links to child generators.
• asyncio, async/await natively supports asynchronous programming.
• New modules: enum, mock, ipaddress, concurrent.futures, asyncio urllib, selector.
• Different enumeration classes cannot be compared.
• Only equal comparisons are allowed within the same enumeration class.
• Usage of enumeration classes (default numbering starts from 1).
• To avoid duplicate enumeration values in the enumeration class, use @unique to decorate the enumeration class.

# Notes on enumerations
from enum import Enum

class COLOR(Enum):
    YELLOW=1
# YELLOW=2 # will raise an error
    GREEN=1 # will not raise an error, GREEN can be seen as an alias for YELLOW
    BLACK=3
    RED=4
print(COLOR.GREEN) # COLOR.YELLOW, will still print YELLOW
for i in COLOR: # Iterating over COLOR will not show GREEN
    print(i)
# COLOR.YELLOW
COLOR.BLACK
COLOR.RED
# How to iterate over aliases
for i in COLOR.__members__.items():
    print(i)
# output: ('YELLOW', <COLOR.YELLOW: 1>)
# ('GREEN', <COLOR.YELLOW: 1>)
# ('BLACK', <COLOR.BLACK: 3>)
# ('RED', <COLOR.RED: 4>)
for i in COLOR.__members__:
    print(i)
# output: YELLOW
GREEN
BLACK
RED

# Enumeration conversion
# It is better to use the numeric value of the enumeration when storing in the database rather than the string name of the label
# Use enumeration classes in the code
a=1
print(COLOR(a)) # output: COLOR.YELLOW

Py2/3 Conversion Tools

• six module: a module compatible with Python 2 and Python 3.

• 2to3 tool: changes code syntax version.

• __future__: uses features from the next version.

Library Related

Common Libraries

• Must-know collections https://segmentfault.com/a/1190000017385799
• Python sorting operations and heapq module https://segmentfault.com/a/1190000017383322
• itertools module super useful methods https://segmentfault.com/a/1190000017416590

Less Common but Important Libraries

• dis (code bytecode analysis)
• inspect (generator state)
• cProfile (performance analysis)
• bisect (maintains ordered lists)
• fnmatch
• fnmatch(string, “*.txt”) # case insensitive under Windows
• fnmatch based on system decision
• fnmatchcase fully case sensitive
• timeit (code execution time)

def isLen(strString):
    # should still use the ternary expression, faster
    return True if len(strString) > 6 else False

def isLen1(strString):
    # note the positions of false and true here
    return [False, True][len(strString) > 6]
import timeit
print(timeit.timeit('isLen1("5fsdfsdfsaf")', setup="from __main__ import isLen1"))

print(timeit.timeit('isLen("5fsdfsdfsaf")', setup="from __main__ import isLen"))

• contextlib
• @contextlib.contextmanager makes generator functions into context managers.
• types (contains all type objects defined by the standard interpreter, can decorate generator functions as asynchronous).

import types
types.coroutine # equivalent to implementing __await__

• html (implementing HTML escaping).

import html
html.escape("<h1>I'm Jim</h1>") # output: '<h1>I'm Jim</h1>'
html.unescape('<h1>I'm Jim</h1>') # <h1>I'm Jim</h1>

• mock (solves test dependencies).
• concurrent (creates process pools and thread pools).

from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor()
task = pool.submit(function_name, (parameters)) # this method will not block, will return immediately
task.done() # check if the task has completed
task.result() # blocking method, check the task return value
task.cancel() # cancel unexecuted tasks, returns True or False, returns True if successful

task.add_done_callback() # callback function
task.running() # whether it is executing task is a Future object

for data in pool.map(function, parameter_list): # returns the list of completed task results, executed in the order of parameters
    print(the result of task completion data)

from concurrent.futures import as_completed
as_completed(task_list) # returns the list of completed tasks, one returns when completed

wait(task_list, return_when=condition) # block the main thread based on conditions, there are four conditions

• selector (encapsulates select, user multiplexing IO programming).
• asyncio

future = asyncio.ensure_future(coroutine) # equivalent to future = loop.create_task(coroutine)
future.add_done_callback() # add a callback function after completion
loop.run_until_complete(future)
future.result() # check the return result

asyncio.wait() accepts an iterable of coroutine objects
asyncio.gather(*iterable_object, *iterable_object) # both results are the same, but gather can cancel in bulk, gather object.cancel()

There is only one loop in a thread

When loop.stop must loop.run_forever() otherwise it will report an error
loop.run_forever() can execute non-coroutines
Finally execute finally module loop.close()

asyncio.Task.all_tasks() gets all tasks and then iterates through and uses task.cancel() to cancel

Partial function partial(function, parameters) wraps the function into another function name, its parameters must be placed in front of defining the function

loop.call_soon(function, parameters)
call_soon_threadsafe() thread safe
loop.call_later(time, function, parameters)
In the same code block call_soon is executed first, then multiple later are executed according to the ascending order of time

If you must run blocking code
Use loop.run_in_executor(executor, function, parameters) to wrap into a multi-threaded task list, run using wait(task list)

Using asyncio to implement http
reader, writer = await asyncio.open_connection(host, port)
writer.writer() send request
async for data in reader:
    data = data.decode("utf-8")
    list.append(data)
Then the list stores the html

as_completed(tasks) returns one when completed, returns an iterable object

Coroutine lock
async with Lock():

Advanced Python

• Inter-process communication:
• Manager (built-in many data structures, can achieve memory sharing between multiple processes).

from multiprocessing import Manager, Process
def add_data(p_dict, key, value):
    p_dict[key] = value

if __name__ == "__main__":
    progress_dict = Manager().dict()
    from queue import PriorityQueue

    first_progress = Process(target=add_data, args=(progress_dict, "bobby1", 22))
    second_progress = Process(target=add_data, args=(progress_dict, "bobby2", 23))

    first_progress.start()
    second_progress.start()
    first_progress.join()
    second_progress.join()

    print(progress_dict)

• Pipe (suitable for two processes).

from multiprocessing import Pipe, Process
# The performance of pipe is higher than queue
def producer(pipe):
    pipe.send("bobby")

def consumer(pipe):
    print(pipe.recv())

if __name__ == "__main__":
    recevie_pipe, send_pipe = Pipe()
    # pipe can only be used for two processes
    my_producer = Process(target=producer, args=(send_pipe, ))
    my_consumer = Process(target=consumer, args=(recevie_pipe,))

    my_producer.start()
    my_consumer.start()
    my_producer.join()
    my_consumer.join()

• Queue (cannot be used for process pools, communication between process pools requires using Manager().Queue()).

from multiprocessing import Queue, Process
def producer(queue):
    queue.put("a")
    time.sleep(2)

def consumer(queue):
    time.sleep(2)
    data = queue.get()
    print(data)

if __name__ == "__main__":
    queue = Queue(10)
    my_producer = Process(target=producer, args=(queue,))
    my_consumer = Process(target=consumer, args=(queue,))
    my_producer.start()
    my_consumer.start()
    my_producer.join()
    my_consumer.join()

• Process Pool

def producer(queue):
    queue.put("a")
    time.sleep(2)

def consumer(queue):
    time.sleep(2)
    data = queue.get()
    print(data)

if __name__ == "__main__":
    queue = Manager().Queue(10)
    pool = Pool(2)

    pool.apply_async(producer, args=(queue,))
    pool.apply_async(consumer, args=(queue,))

    pool.close()
    pool.join()

• Common methods in the sys module

• argv command line argument list, the first is the path of the program itself.
• path returns the module search path.
• modules.keys() returns a list of all imported modules.
• exit(0) exits the program.

• a in s or b in s or c in s shorthand

• Using any: all() returns True for any iterable object that is empty.

# Method 1
True in [i in s for i in [a, b, c]]
# Method 2
any(i in s for i in [a, b, c])
# Method 3
list(filter(lambda x: x in s, [a, b, c]))

• Using set collections

• {1, 2}.issubset({1, 2, 3}) # check if it is a subset.
• {1, 2, 3}.issuperset({1, 2})
• {}.isdisjoint({}) # check if the intersection of two sets is empty, if it is empty it returns True.

• Code matching Chinese characters

• [u4E00-u9FA5] matches the range of Chinese characters [from one to 龥].

• Check the system default encoding format

import sys
sys.getdefaultencoding()    # setdefaultencodeing() sets the system encoding method

• getattr VS getattribute

class A(dict):
    def __getattr__(self, value): # returns when accessing a property that does not exist
        return 2
    def __getattribute__(self, item): # shields all element access
        return item

• Class variables will not be stored in the instance __dict__, but will only exist in the class’s __dict__.

• globals/locals (can manipulate code indirectly).

• globals contains all variable attributes and values in the current module.
• locals contains all variable attributes and values in the current environment.

• Python variable name resolution mechanism (LEGB)

• Local scope (Local).
• The current scope embedded in the local scope (Enclosing locals).
• Global/module scope (Global).
• Built-in scope (Built-in).

• Implement grouping from 1-100 every three.

print([[x for x in range(1, 101)][i:i + 3] for i in range(0, 100, 3)])

• What is a metaclass?
• A class that creates classes; when creating a class, just set metaclass=metaclass, the metaclass needs to inherit from type instead of object, because type is the metaclass.

type.__bases__  # (<class 'object'>,)
object.__bases__    # ()
type(object)    # <class 'type'>

class Yuan(type):
        def __new__(cls, name, base, attr, *args, **kwargs):
            return type(name, base, attr, *args, **kwargs)
    class MyClass(metaclass=Yuan):
        pass

• What is duck typing (i.e., polymorphism)?

• Python does not default to judging parameter types when using passed parameters; as long as the parameters meet execution conditions, they can be executed.

• Deep copy and shallow copy

• Deep copy copies content, shallow copy copies addresses (increases reference count).
• Copy module implements deep copy.

• Unit testing

• Generally, test classes inherit from the TestCase module under unittest.
• pytest module for quick testing (methods start with test_/test files start with test_/test classes start with Test and cannot have init methods).
• coverage statistics test coverage.

class MyTest(unittest.TestCase):
    def tearDown(self): # Executed before each test case
        print('This method has started testing')

    def setUp(self): # Do operations before each test case is executed
        print('This method has finished testing')

    @classmethod
    def tearDownClass(self): # Must use @classmethod decorator, runs once after all tests run
        print('Start testing')
    @classmethod
    def setUpClass(self): # Must use @classmethod decorator, runs once before all tests run
        print('End testing')

    def test_a_run(self):
        self.assertEqual(1, 1)  # Test case

• GIL will release based on the number of executed bytecode lines and time slices; GIL actively releases when encountering IO operations.

• What is monkey patch?

• Monkey patch, replaces blocking syntax with non-blocking methods during runtime.

• What is introspection?

• The ability to determine the type of an object at runtime, id, type, isinstance.

• Is Python pass-by-value or pass-by-reference?

• Neither; Python is shared parameter passing; default parameters will only execute once during execution.

• The difference between else and finally in try-except-else-finally

• else executes when no exception occurs; finally executes regardless of whether an exception occurs.
• except can catch multiple exceptions at once, but generally for different processing of different exceptions, we capture and process separately.

• GIL global interpreter lock

• Only one thread can execute at a time; this is a feature of CPython (IPython), other interpreters do not have this.
• CPU-intensive: multi-process + process pool.
• IO-intensive: multi-threading/coroutines.

• What is Cython?

• A tool that converts Python to C code.

• Generators and iterators

• Objects that implement the __next__ and __iter__ methods are iterators.
• Iterable objects only need to implement the __iter__ method.
• Using generator expressions or yield in generator functions (generators are a special type of iterator).

• What is a coroutine?

• A much lighter multitasking method than threads.
• Implementation methods:
• yield
• async-await

• dict underlying structure

• A hash table is used as the underlying structure to support fast lookups.
• The average lookup time complexity of a hash table is O(1).
• The CPython interpreter uses secondary probing to solve hash collision problems.

• Hash expansion and hash collision resolution solutions

• Loop copying to new space achieves expansion.
• Collision resolution:
• Chaining method.
• Secondary probing (open addressing method): Python uses this.

for gevent import monkey
monkey.patch_all()  # modifies all blocking methods in the code, can specify specific methods to modify

• Check if it is a generator or coroutine

co_flags = func.__code__.co_flags

# Check if it is a coroutine
if co_flags & 0x180:
    return func

# Check if it is a generator
if co_flags & 0x20:
    return func

• Fibonacci problem and its variations

# A frog can jump up 1 step or 2 steps at a time. How many ways can the frog jump up a staircase of n steps?
# How many ways can n 2*1 small rectangles cover a 2*n large rectangle without overlapping?
# Method 1:
fib = lambda n: n if n <= 2 else fib(n - 1) + fib(n - 2)
# Method 2:
def fib(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return b

# A frog can jump up 1 step or 2 steps... it can also jump up n steps. How many ways can the frog jump up a staircase of n steps?
fib = lambda n: n if n < 2 else 2 * fib(n - 1)

• Get the environment variable set on the computer

import os
os.getenv(env_name, None) # get environment variable, if it does not exist return None

• Garbage collection mechanism
• Reference counting.
• Mark clearing.
• Generational collection.

# Check the trigger of generational collection
import gc
gc.get_threshold()  # output: (700, 10, 10)

• True and False are completely equivalent to 1 and 0 in code, can be calculated directly with numbers; inf represents infinity.

• C10M/C10K

• C10M: 8-core CPU, 64GB memory, maintaining 10 million concurrent connections on a 10gbps network.
• C10K: 1GHz CPU, 2GB memory, maintaining 10,000 clients providing FTP services on a 1gbps network.

• The difference between yield from and yield:

• yield from follows an iterable object, while yield has no restrictions.
• GeneratorExit is triggered when the generator stops.

• Several uses of a single underscore

• When defining a variable, it indicates a private variable.
• When unpacking, it indicates discarding useless data.
• In interactive mode, it indicates the result of the last executed code.
• Can be used for number concatenation (111_222_333).

• Using break will not execute else.

• Decimal to binary conversion

def conver_bin(num):
    if num == 0:
        return num
    re = []
    while num:
        num, rem = divmod(num, 2)
        re.append(str(rem))
    return "".join(reversed(re))
  conver_bin(10)

• list1 = [‘A’, ‘B’, ‘C’, ‘D’] how to get a new list named after the elements in the list A=[], B=[], C=[], D=[]?

list1 = ['A', 'B', 'C', 'D']

# Method 1
for i in list1:
    globals()[i] = []   # Can be used to implement Python reflection

# Method 2
for i in list1:
    exec(f'{i} = []')   # exec executes string statement

• memoryview and bytearraynotcommon,justnoteditdown

# bytearray is mutable, bytes is immutable, memoryview does not produce new slices and objects
a = 'aaaaaa'
ma = memoryview(a)
ma.readonly  # read-only memoryview
mb = ma[:2]  # will not produce new string

a = bytearray('aaaaaa')
ma = memoryview(a)
ma.readonly  # writable memoryview
mb = ma[:2]      # will not produce new bytearray
mb[:2] = 'bb'    # changes to mb are changes to ma

• Ellipsis type

# The phenomenon of ... (ellipsis) appearing in code is an Ellipsis object
L = [1, 2, 3]
L.append(L)
print(L)    # output: [1, 2, 3, [...]]

• Lazy evaluation

class lazy(object):
    def __init__(self, func):
        self.func = func

    def __get__(self, instance, cls):
        val = self.func(instance)    # which is equivalent to executing area(c), c is the Circle object below
        setattr(instance, self.func.__name__, val)
        return val

class Circle(object):
    def __init__(self, radius):
        self.radius = radius

    @lazy
    def area(self):
        print('evaluate')
        return 3.14 * self.radius ** 2

• Traverse files, pass in a folder, and print out all file paths inside (recursive).

all_files = []    
def getAllFiles(directory_path):
    import os                                       
    for sChild in os.listdir(directory_path):                
        sChildPath = os.path.join(directory_path, sChild)
        if os.path.isdir(sChildPath):
            getAllFiles(sChildPath)
        else:
            all_files.append(sChildPath)
    return all_files

• File storage, processing of file names

# secure_filename converts the string into a safe file name
from werkzeug import secure_filename
secure_filename("My cool movie.mov") # output: My_cool_movie.mov
secure_filename("../../../etc/passwd") # output: etc_passwd
secure_filename(u'i contain cool mluts.txt') # output: i_contain_cool_umlauts.txt

• Date formatting

from datetime import datetime

datetime.now().strftime("%Y-%m-%d")

import time
# Only localtime can be formatted, time cannot be formatted
# time.strftime("%Y-%m-%d", time.localtime())

• strange issue with tuple using +=

# Will raise an error, but the value of the tuple will change because t[1] id has not changed
t = (1, [2, 3])
t[1] += [4, 5]
# t[1] using append/extend methods will not raise an error and can execute successfully

• __missing__ you should know

class Mydict(dict):
    def __missing__(self, key): # returns the value when Mydict uses slice access and the property does not exist
        return key

• + and +=

# + cannot be used to connect lists and tuples, while += can (via iadd implementation, the internal implementation method is extends(), so it can increase tuples), + will create new objects
# Immutable objects do not have the __iadd__ method, so they directly use the __add__ method, so tuples can use += for addition between tuples

• How to turn each element of an iterable into all keys of a dictionary?

dict.fromkeys(['jim', 'han'], 21) # output: {'jim': 21, 'han': 21}

Network Knowledge

• What is HTTPS?

• Secure HTTP protocol; HTTPS requires a CS certificate, data encryption, port 443, secure; the same website with HTTPS SEO ranking will be higher.

• Common response status codes

204 No Content // Request processed successfully, no entity body returned, generally used to indicate successful deletion.
206 Partial Content // Get range request has been successfully processed.
303 See Other // Temporary redirect, expected to use GET to redirect.
304 Not Modified // Request cached resources.
307 Temporary Redirect // Temporary redirect, Post will not become Get.
401 Unauthorized // Authentication failed.
403 Forbidden // Resource request denied.
400 // Request parameter error.
201 // Add or change successfully.
503 // Server maintenance or overload.

• Idempotency and security of HTTP request methods

• WSGI

# environ: a dict object containing all HTTP request information
# start_response: a function to send HTTP response
def application(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/html')])
    return '<h1>Hello, web!</h1>'

• RPC

• CDN

• SSL (Secure Sockets Layer) and its successor Transport Layer Security (TLS) are security protocols that provide security and data integrity for network communications.

• SSH (Secure Shell Protocol) is a security protocol established by the IETF Network Working Group; SSH is a security protocol built on the application layer. SSH is currently a reliable protocol designed for remote login sessions and other network services. Using the SSH protocol can effectively prevent information leakage during remote management. SSH was originally a program on UNIX systems and has rapidly expanded to other operating platforms. When used correctly, SSH can compensate for vulnerabilities in the network. SSH clients are available for multiple platforms. Almost all UNIX platforms—including HP-UX, Linux, AIX, Solaris, Digital UNIX, Irix—can run SSH.

• TCP/IP

• Although theoretically, once all four packets are sent, we can directly enter the CLOSE state, we must pretend that the network is unreliable, and there may be a last ACK loss. So the TIME_WAIT state is used to resend possibly lost ACK packets.
• When the server receives the client’s SYN connection request packet, it can directly send a SYN+ACK packet. The ACK packet is used for acknowledgment, and the SYN packet is used for synchronization. However, when closing the connection, when the server receives the FIN packet, it may not immediately close the SOCKET, so it can only reply with an ACK packet to tell the client, ‘I received the FIN packet you sent.’ Only after all packets sent by the server are completed can it send a FIN packet, so they cannot be sent together. Hence the need for a four-way handshake.
• Three-way handshake (SYN/SYN+ACK/ACK).
• Four-way handshake (FIN/ACK/FIN/ACK).
• TCP: connection-oriented/reliable/byte stream-based.
• UDP: connectionless/unreliable/message-oriented.
• Three-way handshake and four-way handshake.
• Why is there a three-way handshake when connecting, but four when closing?
• Why does the TIME_WAIT state need to go through 2MSL (maximum segment lifetime) to return to the CLOSE state?

• XSS/CSRF

• HttpOnly prohibits JS scripts from accessing and manipulating cookies, effectively preventing XSS.

MySQL

• Index improvement process

• Linear structure -> binary search -> hash -> binary search tree -> balanced binary tree -> multi-way search tree -> multi-way balanced search tree (B-Tree).

• MySQL interview summary basic edition

• https://segmentfault.com/a/1190000018371218

• MySQL interview summary advanced edition

• https://segmentfault.com/a/1190000018380324

• In-depth understanding of MySQL

• http://ningning.today/2017/02/13/database/深入浅出mysql/

• When clearing an entire table, InnoDB deletes one row at a time, while MyISAM will delete and recreate the table.

• text/blob data types cannot have default values, and there is no case conversion when querying.

• When does an index fail?

• Avoid using != or <> operators in the where clause, otherwise the engine will give up using the index and perform a full table scan.
• Avoid using or to connect conditions in the where clause, otherwise it will cause the engine to give up using the index and perform a full table scan, even if some conditions have indexes; this is also why using or should be minimized.
• If the column type is a string, it must be quoted in the condition, otherwise the index will not be used.
• Avoid performing function operations on fields in the where clause, which will cause the engine to give up using the index and perform a full table scan.
• For multi-column indexes, if the first part is not used, the index will not be used.
• Like fuzzy queries starting with %.
• Implicit type conversion occurs.
• The leftmost prefix rule is not satisfied.
• Failure scenarios:

For example:
select id from t where substring(name, 1, 3) = 'abc' – name;
should be changed to:
select id from t where name like 'abc%'
For example:
select id from t where datediff(day, createdate, '2005-11-30') = 0 – '2005-11-30';
should be changed to:

Do not perform function, arithmetic operations, or other expression operations on the left side of ‘=’ in the where clause, otherwise the system may not be able to correctly use the index.

Avoid performing expression operations on fields in the where clause, which will cause the engine to give up using the index and perform a full table scan.

For example:
select id from t where num/2 = 100
should be changed to:
select id from t where num = 100*2;

Not suitable for columns with fewer key values (columns with more duplicate data), for example: set enum columns are not suitable (enumeration type (enum) can add null, and the default value will automatically filter the space set (set) and enumeration is similar, but can only add 64 values).

If MySQL estimates that a full table scan will be faster than using an index, it will not use the index.

• What is a clustered index?
• B+Tree leaf nodes save data or pointers.
• MyISAM index and data are separated, using a non-clustered index.
• InnoDB data files are index files, and primary key indexes are clustered indexes.

Redis Command Summary

• Why is it so fast?

• Because Redis operates based on memory, the CPU is not the bottleneck of Redis; the bottleneck is most likely the size of the machine’s memory or network bandwidth. Since single-threading is easy to implement, and the CPU will not become a bottleneck, it is logical to adopt a single-threaded solution (after all, adopting multi-threading would involve many troubles!).
• Based in memory, written in C language.
• Uses multiplexed I/O model, non-blocking IO.
• Uses single-threading to reduce context switching.
• Simple data structure.
• Self-built VM mechanism reduces the time for calling system functions.

• Advantages

• High performance – Redis can read at a speed of 110,000 times/s, and write at a speed of 81,000 times/s.
• Rich data types.
• Atomic – all operations of Redis are atomic, and Redis also supports atomic execution of several operations combined.
• Rich features – Redis also supports publish/subscribe, notifications, key expiration, and other features.

• What is a Redis transaction?

• A mechanism that packages multiple requests and executes multiple commands once and in order.
• Implement transaction functions through multi, exec, watch, etc.
• Python redis-py pipeline=conn.pipeline(transaction=True).

• Persistence methods

• save (synchronous, can ensure data consistency).
• bgsave (asynchronous, used by default without AOF during shutdown).
• RDB (snapshot).
• AOF (append log).

• How to implement a queue

• push.
• rpop.

• Common data types (Bitmaps, Hyperloglogs, range queries are not commonly used)

• skiplist (skip list).
• intset or hashtable.
• ziplist (continuous memory block, each entry node head saves information about the length of the previous and next nodes to implement a doubly linked list function) or double linked list.
• Integer or sds (Simple Dynamic String).
• String: counter.
• List: user’s followers, fans list.
• Hash: user followers.
• Zset: real-time information leaderboard.

• Differences with Memcached

• Memcached can only store string keys.
• Memcached users can only add data to the end of existing strings using the APPEND method, treating the string as a list. However, when deleting these elements, Memcached uses a blacklist method to hide elements in the list, thereby avoiding operations such as reading, updating, and deleting elements.
• Both Redis and Memcached store data in memory and are in-memory databases. However, Memcached can also be used to cache other things, such as images, videos, etc.
• Virtual memory – Redis can swap some rarely used values to disk when physical memory runs out.
• Data storage security – when Memcached crashes, the data is lost; Redis can periodically save to disk (persistence).
• Different application scenarios: Redis can be used as a NoSQL database, as well as a message queue, data stack, and data cache; Memcached is suitable for caching SQL statements, data sets, user temporary data, delayed query data, and sessions.

• Implementing distributed locks with Redis

• Using setnx to implement locking, can also add an expiration time through expire.
• The value of the lock can be a random uuid or a specific name.
• When releasing the lock, check if it is the lock using uuid, if so, execute delete to release the lock.

• Common issues

• When access volume surges, services encounter issues (such as slow response time or unresponsiveness) or non-core services affect the performance of core processes, it is still necessary to ensure that services are available, even if it is a degraded service. The system can automatically downgrade based on some key data or can be configured to implement manual downgrading.
• Data expiration, update cached data.
• Initializing the project, adding some commonly used data to the cache.
• When requesting access data, querying the cache does not exist, and it does not exist in the database either.
• Short-term cache data expiration, a large number of requests accessing the database.
• Cache avalanche.
• Cache penetration.
• Cache warming.
• Cache updates.
• Cache downgrading.

• Consistent Hash algorithm

• Used to ensure data consistency when using clusters.

• Implement a distributed lock based on Redis, requiring a timeout parameter.

• setnx.

• Virtual memory.

• Memory jitter.

Linux

• Five I/O models of Unix

• select.
• poll.
• epoll.
• Not high concurrency, in cases with very active connection numbers.
• Not much improvement over select.
• Suitable for cases with a large number of connections but few active links.
• Blocking IO.
• Non-blocking IO.
• Multiplexing IO (implemented using selectot in Python).
• Signal-driven IO.
• Asynchronous IO (Gevent/Asyncio implement asynchronous).

• Command manual better than man

• tldr: a manual with command examples.

• Difference between kill -9 and -15

• -15: program stops immediately/after the program releases the corresponding resources, stops/ the program may still continue to run.
• -9: due to the uncertainty of -15, directly use -9 to immediately kill the process.

• Paging mechanism (logical and physical address separation memory allocation management scheme):

• The operating system manages memory efficiently to reduce fragmentation.
• The logical address of the program is divided into fixed-size pages.
• The physical address is divided into frames of the same size.
• Through the page table corresponding logical address and physical address.

• Segmentation mechanism

• To meet some logical requirements of code.
• Data sharing/data protection/dynamic linking.
• Each segment internally is continuously allocated memory, while segments are allocated discretely.

• How to check CPU memory usage?

• top.
• free to check available memory, troubleshoot memory leaks.

Design Patterns

Singleton Pattern

# Method 1
def Single(cls, *args, **kwargs):
    instances = {}
    def get_instance(*args, **kwargs):
        if cls not in instances:
            instances[cls] = cls(*args, **kwargs)
        return instances[cls]
    return get_instance
@Single
class B:
    pass
# Method 2
class Single:
    def __init__(self):
        print("Singleton pattern implementation method two...")

single = Single()
del Single  # Every time you call single you can do it.
# Method 3 (most commonly used method)
class Single:
    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, '_instance'):
            cls._instance = super().__new__(cls, *args, **kwargs)
        return cls._instance

Factory Pattern

class Dog:
    def __init__(self):
        print("Wang Wang Wang")
class Cat:
    def __init__(self):
        print("Miao Miao Miao")

def fac(animal):
    if animal.lower() == "dog":
        return Dog()
    if animal.lower() == "cat":
        return Cat()
    print("Sorry, must be: dog, cat")

Builder Pattern

class Computer:
    def __init__(self, serial_number):
        self.serial_number = serial_number
        self.memory = None
        self.hadd = None
        self.gpu = None
    def __str__(self):
        info = (f'Memory: {self.memoryGB}',
        'Hard Disk: {self.hadd}GB',
        'Graphics Card: {self.gpu}')
        return ''.join(info)
class ComputerBuilder:
    def __init__(self):
        self.computer = Computer('Jim1996')
    def configure_memory(self, amount):
        self.computer.memory = amount
        return self # For convenient chaining calls
    def configure_hdd(self, amount):
        pass
    def configure_gpu(self, gpu_model):
        pass
class HardwareEngineer:
    def __init__(self):
        self.builder = None
    def construct_computer(self, memory, hdd, gpu):
        self.builder = ComputerBuilder()
        self.builder.configure_memory(memory).configure_hdd(hdd).configure_gpu(gpu)
    @property
    def computer(self):
        return self.builder.computer

Data Structures and Algorithms

Implement various data structures in Python

Quick Sort

def quick_sort(_list):
    if len(_list) < 2:
        return _list
    pivot_index = 0
    pivot = _list[pivot_index]
    left_list = [i for i in _list[:pivot_index] if i < pivot]
    right_list = [i for i in _list[pivot_index:] if i > pivot]
    return quick_sort(left_list) + [pivot] + quick_sort(right_list)

Selection Sort

def select_sort(seq):
    n = len(seq)
    for i in range(n - 1):
        min_idx = i
        for j in range(i + 1, n):
            if seq[j] < seq[min_idx]:
                min_idx = j
        if min_idx != i:
            seq[i], seq[min_idx] = seq[min_idx], seq[i]

Insertion Sort

def insertion_sort(_list):
    n = len(_list)
    for i in range(1, n):
        value = _list[i]
        pos = i
        while pos > 0 and value < _list[pos - 1]:
            _list[pos] = _list[pos - 1]
            pos -= 1
        _list[pos] = value
        print(sql)

Merge Sort

def merge_sorted_list(_list1, _list2):   # Merge ordered lists
    len_a, len_b = len(_list1), len(_list2)
    a = b = 0
    sort = []
    while len_a > a and len_b > b:
        if _list1[a] > _list2[b]:
            sort.append(_list2[b])
            b += 1
        else:
            sort.append(_list1[a])
            a += 1
    if len_a > a:
        sort.append(_list1[a:])
    if len_b > b:
        sort.append(_list2[b:])
    return sort

def merge_sort(_list):
    if len(_list) < 2:
        return _list
    else:
        mid = int(len(_list) / 2)
        left = merge_sort(_list[:mid])
        right = merge_sort(_list[mid:])
        return merge_sorted_list(left, right)

Heap Sort using heapq module

from heapq import nsmallest
def heap_sort(_list):
    return nsmallest(len(_list), _list)

Stack

from collections import deque
class Stack:
    def __init__(self):
        self.s = deque()
    def peek(self):
        p = self.pop()
        self.push(p)
        return p
    def push(self, el):
        self.s.append(el)
    def pop(self):
        return self.s.pop()

Queue

from collections import deque
class Queue:
    def __init__(self):
        self.s = deque()
    def push(self, el):
        self.s.append(el)
    def pop(self):
        return self.s.popleft()

Binary Search

def binary_search(_list, num):
    mid = len(_list) // 2
    if len(_list) < 1:
        return False
    if num > _list[mid]:
        return binary_search(_list[mid:], num)
    elif num < _list[mid]:
        return binary_search(_list[:mid], num)
    else:
        return _list.index(num)

Interview Enhancement Questions

• About database optimization and design
• Use hash consistency algorithm.
• setnx.
• setnx + expire.
• Use Redis.
• If the data writing order of the InnoDB table can be consistent with the order of the B+ tree index leaf nodes, the storage and access efficiency will be the highest. To optimize storage and query performance, self-incrementing IDs should be used as the primary key.
• For the primary index of InnoDB, data will be sorted according to the primary key. Due to the unordered nature of UUID, InnoDB will produce huge IO pressure, so it is not suitable to use UUID as a physical primary key; it can be used as a logical primary key, while the physical primary key still uses self-incrementing ID. To ensure global uniqueness, UUID should be used to index and associate other tables or as foreign keys.
• https://segmentfault.com/a/1190000018426586
• How to implement a queue using two stacks.
• Reverse linked list.
• Merge two ordered linked lists.
• Delete linked list nodes.
• Reverse binary tree.
• Design a short URL service? 62-base implementation.
• Design a flash sale system (feed stream)?
• https://www.jianshu.com/p/ea0259d109f9
• Why is it better to use self-incrementing integers for MySQL database primary keys? Can UUID be used? Why?
• If it is a distributed system, how do we generate self-incrementing IDs for the database?
• Implement a distributed lock based on Redis, requiring a timeout parameter.
• If a single Redis node goes down, how to handle it? Are there other industry solutions to implement distributed lock codes?

Cache Algorithms

• LRU (least-recently-used): replaces the least recently used objects.

• LFU (Least frequently used): the least frequently used; if a piece of data has been used very few times in a recent period, it is also unlikely to be used in the future.

Server Performance Optimization Directions

• Use data structures and algorithms.

• Database

• Enable slow_query_log_file and query slow query logs.
• Use explain to troubleshoot index issues.
• Adjust data modification indexes.
• Index optimization.
• Eliminate slow queries.
• Batch operations to reduce IO operations.
• Use NoSQL: such as Redis.

• Network IO

• Batch operations.
• Pipeline.

• Cache

• Redis.

• Asynchronous

• Asyncio for asynchronous operations.
• Use Celery to reduce IO blocking.

• Concurrency

• Multithreading.

• Gevent.

How to obtain: