MAC下完成人脸检测代码环境构建

目的

工作中用到图片的截图,但在使用过程中出现了一个尴尬的问题,就是截图时,有的人没有了头
,只留下身子.

解决方式

通过代码来检测出头部所以位置,然后来决定载哪些区域(目前,只是对一个人进行剪切。

具体方案

  1. 找到一个合适的类库来进行人头位置检测

     _python-opencv_
  2. 使用python 对人裁剪范围进行处理(GraphicsMagick)得到想要的区域

正题 如何在MAC上安装开发环境

利用 python-opencv 库
http://www.pyimagesearch.com/2015/06/15/install-opencv-3-0-and-python-2-7-on-osx/


cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local \
-D PYTHON2_PACKAGES_PATH=~/.virtualenvs/cv/lib/python2.7/site-packages \
-D
PYTHON2_LIBRARY=/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/bin
\
-D PYTHON2_INCLUDE_DIR=/usr/local/Frameworks/Python.framework/Headers \
-D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON \
-D BUILD_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules ..

http://docs.opencv.org/2.4/doc/tutorials/introduction/linux_install/linux_install.html#linux-installation


brew tap homebrew/science
brew install opencv3

http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html

图片处理环境构建

问题

对存储的文件进行并发处理,由于处理的图片量比较大。
图片处理又比较耗内存,为防止影响正常的服务,故将其与业务服务分开部署

方案

python+uwsgi+imagemagick

环境安装:


1.配置
16Core,16G内存,50G硬盘
2.端口
13814(fcp),8010(http)
3.安装 graphicsmagick
jasper-1.900.1.zip jpegsrc.v9a.tar.gz

wget http://www.imagemagick.org/download/delegates/jpegsrc.v9a.tar.gz
wget http://www.ece.uvic.ca/~frodo/jasper/software/jasper-1.900.1.zip
gm 命令ln /usr/local/graphicsmagick/bin/gm usr/bin/gm

4.装音频插件和字体
http://my.oschina.net/ethan09/blog/372435?fromerr=WsZSucMx
安装微软雅黑字体

5.python版本升级
http://blog.csdn.net/jcjc918/article/details/11022345 由于django必须要用2.7,所以这里要进行升级
注意修改完后 要修改/usr/bin/supervisord,/usr/bin/supervisorctl 的头为/usr/bin/python2.6

6.重新安装pip和c++库
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py

安装c++库
sudo yum install GraphicsMagick-c++-devel
sudo yum install boost-devel
7.安装python插件

sudo pip install -r requirements.txt
requirements.txt内容
Django==1.9.7
beautifulsoup4==4.4.1
protobuf==2.6.1
pgmagick==0.6.2
uWSGI==2.0.13.1
wsgiref==0.1.2
django-log-request-id==1.0.0

8.安装nginx
yum install nginx
9.安装uwsgi
sudo yum install uwsgi
sudo yum install uwsgi-plugin-python

nginx配置

location / {
uwsgi_pass unix:///tmp/uwsgi.sock;
include /etc/nginx/uwsgi_params; # the uwsgi_params file you installed
}

client_max_body_size 20M;

10.安装图片处理服务
cd /opt/fs/
git clone http://git.firstshare.cn/Qixin/FSPythonWSGIProcess.git
cd FSPythonWSGIProcess

uwsgi --ini online_uwsgi.ini --buffer-size 20971520 --daemonize ./logs/uwsgi.log

11.异常
*** Starting uWSGI 2.0.12 (64bit) on [Tue Jun 14 16:36:20 2016] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-16) on 02 January 2016 19:53:13
os: Linux-2.6.32-573.8.1.el6.x86_64 #1 SMP Tue Nov 10 18:01:38 UTC 2015
nodename: vlnx160170.fsceshi.com
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /home/wans/FSPythonWSGIProcess
writing pidfile to /tmp/FsPythonWSGIProcess.pid
detected binary path: /usr/sbin/uwsgi
your processes number limit is 1024
your memory page size is 4096 bytes
detected max file descriptor number: 60000
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 4
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 1476277 bytes (1441 KB) for 10 cores
*** Operational MODE: preforking ***
*** no app loaded. going in full dynamic mode ***

关闭nginx ,启动uwsgi后再启动nginx

优化点:
1.调整nginx
worker_processes 4;
worker_cpu_affinity 1000 0100 0010 0001;
2.#cluster
upstream backend{
server unix:///tmp/uwsgi.sock;
server unix:///tmp/uwsgi1.sock;
server unix:///tmp/uwsgi2.sock;
server unix:///tmp/uwsgi3.sock;
}

location / {
include uwsgi_params;
uwsgi_pass backend;
}

问题总结:
1.yum安装遇到如下类似问题:
yum install uwsgi
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.btte.net
* extras: mirrors.btte.net
* updates: mirrors.163.com
No package uwsgi available.
Error: Nothing to do
解决方案,添加epl至yum的源即可:
yum install http://mirrors.isu.net.sa/pub/fedora/fedora-epel/7/x86_64/e/epel-release-7-6.noarch.rpm
参考文档:https://fedoraproject.org/wiki/EPEL/zh-cn


将amr,caf转mp3

需求


公司原有文件存储现在要用java进行重构,其中涉及到Android和ios上的音频问题。

  1. Android录音是格式是amr,在电脑上一般是播放不出来的,必须要进行转码。

  2. iPhone录音传上来的是caf,在android上是不能进行播放的。

由于上面的问题,我们要进行文件的转换,另外还有一个需求就是,amr转wav ,这里我们将amr和caf统一转换为map3,另外提供一个接口进行amr2wav的转换。 在网上参考了许多,总结下来还都是调用ffmpeg 进行想着音频的转换。总也来说比较简单。这里时间的关系 ,也不可能去研究底层编解码的东西。


开工

安装ffmpeg

查看当前ffmpeg对mp3的编解码支持情况

    ffmpeg -codecs|grep mp3                                        

     D.A.L. mp3                  MP3 (MPEG audio layer 3) (decoders: mp3 mp3float )
     D.A.L. mp3adu               ADU (Application Data Unit) MP3 (MPEG audio layer 3) (decoders: mp3adu mp3adufloat )
     D.A.L. mp3on4               MP3onMP4 (decoders: mp3on4 mp3on4float )



     

问题

  • ffmpeg: error while loading shared libraries: libavdevice.so.53: cannot open shared object file: No such file or directory
      ffmpeg正常安装后执行ffmpeg时出现如下错误:

      解决办法:

      vi /etc/ld.so.conf

      加入:/usr/local/lib

      执行ldconfig
  • ffmpeg -i test.amr test.mp3
    ffmpeg: error while loading shared libraries: libavdevice.so.56: cannot open shared object file: No such file or directory

     <http://www.tjcarroll.org/?p=51>

Django,Mysql,空间数据是通过Google地图和高德地图进行采集的

##环境

Django,Mysql,空间数据是通过Google地图和高德地图进行采集的

$ mysql -V
mysql Ver 14.14 Distrib 5.6.20, for osx10.9 (x86_64) using EditLine wrapper

##需求描述

mysql中存储了一些地理空间点类型的数,要进行周边查询。

##MySQL空间相关的局限性

MySQL空间扩展的功能仅支持包络框相关的操作(MySQL称之为最小边框,或简称为 MBR)。也就是说,MySQL符合OGC标准。

目前,MySQL没有实现Contains,Crosses,Disjoint,Intersects,Overlaps,Touches函数,可以通过MBR来实现同样效果操作。

也就是说,在MySQL进行如contains类似的空间查询时,可以通过bbcontains来实现同样效果的操作。

注意:

只有MyISAM引擎的MySQL表才真正的支持空间索引(R-trees)。也就是说,当你要使用MySQL提供的空间扩展时,你要在快速查询空间数据和数据的完整性之间做一个选择 - MyISAM的表不支持事务和外键约束。

##数据库配置

空间表引擎 Engine:InnoDB

###创建数据库

mysql>GRANT ALL PRIVILEGES ON *.* TO project_test@localhost IDENTIFIED BY 'project_test' WITH GRANT OPTION; 
mysql>GRANT ALL PRIVILEGES ON *.* TO project_test@"%" IDENTIFIED BY 'project_test' WITH GRANT OPTION;
  • 第一句增加了一个 project_test 用户授权通过本地机(localhost)访问,密码“project_test”。

  • 第二句则是授与 project_test 用户从任何其它主机发起的访问(通配符%)。

###配置数据源

'default': {
'ENGINE': 'django.contrib.gis.db.backends.mysql',
'NAME': 'project_test', # Or path to database file if using sqlite3.
'USER': 'project_test',
'PASSWORD': 'project_test',
'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
'PORT': '3306', # Set to empty string for default.
},

##数据模型的构建

from django.contrib.gis.db import models as gismodels
class AppPoint(gismodels.Model):
description = gismodels.TextField(verbose_name=_(u"描述信息"), max_length=500,
blank=True, null=True)

point = gismodels.PointField(spatial_index=False)

objects = gismodels.GeoManager()

Django Geographic framework 1.7

GeoDjango打算做世界级的地理学Web框架。它的目标是尽可能方便是的利用强大空间数据构建GIS Web 应用。

###GeoQuerySet API

class GeoQuerySet([model=None])

####空间查询

正如使用QuerySet API时一样,在过滤器链(chaining filters)上加上GeoQuerySet进行筛选。除了通常的字段(Field lookups)查询,它还提供了空间字段GeometryField的查询。

可以在这里查看空间查询介绍

下面Django对不同数据库 空间查询操作支持统计表:

    Lookup Type            PostGIS    Oracle    MySQL [7]    SpatiaLite

    bbcontains            X                X             X
    bboverlaps            X                 X             X
    contained            X                 X             X
    contains            X         X        X             X
    contains_properly    X               
    coveredby            X         X          
    covers                X         X          
    crosses                X                               X
    disjoint            X         X        X             X
    distance_gt            X         X                      X
    distance_gte        X         X                      X
    distance_lt            X         X                      X
    distance_lte           X         X                      X
    dwithin                X         X          
    equals                X         X        X             X
    exact                X         X        X             X
    intersects            X         X        X             X
    overlaps            X         X        X             X
    relate                X         X                     X
    same_as                X         X        X             X
    touches                X         X        X             X
    within                X         X        X             X
    left                X               
    right                X               
    overlaps_left       X               
    overlaps_right      X               
    overlaps_above      X               
    overlaps_below      X               
    strictly_above      X               
    strictly_below      X               

####我这里只关注一下对mysql的空间操作支持

按我们的需要我们选用 within

  1. bbcontains

    支持:PostGIS,MySQL,SpatiaLite

    查询数据库中空间数据的bbox包含在指定的空间bbox内的数据。

     数据库         操作  
    
     PostGIS       poly ~ geom
     MySQL         MBRContains(poly,geom)
     SpatiaLite    MbrContains(poly,geom)
  1. bboverlaps

    支持:PostGIS,MySQL,SpatiaLite

    查询数据库中空间数据的bbox与指定的空间bbox相交的数据。

     数据库         操作  
    
     PostGIS       poly && geom
     MySQL         MBROverlops(poly,geom)
     SpatiaLite    MbrOverlops(poly,geom)
  2. contained

    支持:PostGIS,MySQL,SpatiaLite

    查询数据库中空间数据的bbox完全包含指定的空间bbox的数据。

     数据库         操作  
    
     PostGIS       poly @ geom
     MySQL         MBRWithin(poly,geom)
     SpatiaLite    MbrWithin(poly,geom)
  3. contains

    支持:PostGIS,Oracle,MySQL,SpatiaLite

    Example:

     Zipcode.objects.filter(poly__contains=geom)

    查询数据库中空间数据包含指定的空间图形的数据。

     数据库         操作  
    
     PostGIS       ST_Contains(poly, geom)
     Oracle        SDO_CONTAINS(poly, geom)
     MySQL         MBRContains(poly, geom)
     SpatiaLite    Contains(poly, geom)
  1. disjoint

    支持:PostGIS,Oracle,MySQL,SpatiaLite

    Example:

     Zipcode.objects.filter(poly__disjoint=geom)

    查询数据库中与指定的空间图形相离的空间数据。

     数据库         操作  
    
     PostGIS       ST_Disjoint(poly, geom)
     Oracle        SDO_GEOM.RELATE(poly, geom)
     MySQL         MBRDisjoint(poly, geom)
     SpatiaLite    Disjoint(poly, geom)
  2. equals

    支持:PostGIS,Oracle,MySQL,SpatiaLite

  3. exact,same_as

    支持:PostGIS,Oracle,MySQL,SpatiaLite

  4. intersects

    支持:PostGIS,Oracle,MySQL,SpatiaLite

    查询数据库中与指定的空间图形相交的空间数据。

    Example:

         Zipcode.objects.filter(poly__intersects=geom)
        数据库         操作  

        PostGIS       ST_Intersects(poly, geom)
        Oracle        SDO_OVERLAPBDYINTERSECT(poly, geom)
        MySQL         MBRIntersects(poly, geom)
        SpatiaLite    Intersects(poly, geom)
  1. overlaps

    支持:PostGIS,Oracle,MySQL,SpatiaLite

  1. touches

    支持:PostGIS,Oracle,MySQL,SpatiaLite

    Example:

        Zipcode.objects.filter(poly__touches=geom)

    查询与指定的空间几何图形相接的数据。

        数据库         操作  
    
        PostGIS       ST_Touches(poly, geom)
        Oracle        SDO_TOUCH(poly, geom)
        MySQL         MBRTouches(poly, geom)
        SpatiaLite    Touches(poly, geom)
  2. within

    支持:PostGIS,Oracle,MySQL,SpatiaLite

    Example:

        Zipcode.objects.filter(poly__within=geom)

    查询包含在指定的空间几何图形中的数据。

        数据库         操作  
    
        PostGIS       ST_Within(poly, geom)
        Oracle        SDO_INSIDE(poly, geom)
        MySQL         MBRWithin(poly, geom)
        SpatiaLite    Within(poly, geom)

现在知道了要用 within 来查询数据,另一个问题来了,如何生成半径大小为R中心坐标为(x,y)的geom呢。

####创建空间几何图形

可以通过多种方式创建GeosGeometry。第一种方法,就是通过一些参数直接实例化。

  • 下面是分别通过WKT,HEX,WKB和GeoJSON方式直接创建 Geometry 的方法:
In [30]: pnt = GEOSGeometry('POINT(5 23)')

In [31]: pnt = GEOSGeometry('010100000000000000000014400000000000003740')

In [32]: pnt = GEOSGeometry(buffer('\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14@\x00\x00\x00\x00\x00\x007@'))

In [33]: pnt = GEOSGeometry('{ "type": "Point", "coordinates": [ 5.000000, 23.000000 ] }') # GeoJSON

  • 另一种方式就是通过特定类型的空间几何对象的构造器来进行创建该类型的Geometry实例
In [34]: from django.contrib.gis.geos import Point

In [35]: pnt = Point(5,23)

In [36]: pnt
Out[36]: <Point object at 0x10735bb50>

  • 最后一种方法就是通过 fromstr()fromfile 工厂方法来创建Geometry实例。它们分别接收字符串或文件
In [37]: from django.contrib.gis.geos import fromstr,fromfile

In [38]: pnt = fromstr('POINT(5 23)')

In [39]: pnt = fromfile('/path/to/pnt.wkt')

In [34]: pnt = fromfile(open('/path/to/pnt.wkt'))

####实现查询周边几何点的功能

通过上面的学习,在Django中实现mysql数据的周边查询只能通过模糊的查询,
我们这里通过构建一个包络框进行模糊查询:

  • 构建一个包络框
from django.contrib.gis.geos import (Polygon,Point)

point = Point(130,39)

buffer=point.buffer(degree)

  • 进行within查询
AppPoint.objects.filter(point__within=buffer)
  • 问题

这里给的半径通常是米为km,但是这个构建buffer的方法需要的参数是一个度。

degree=l*180/(math.pi*6371)

##测试方法和数据

def get_point(point,r):
EARTH_R=6378.137
buffer = point.buffer(r*180/(math.pi*EARTH_R))
aps=AppPoint.objects.filter(point__within=buffer)
for ap in aps:
print ap.point.json,(math.pi*EARTH_R*ap.point.distance(point)/180)

其中点与点间的距离方法distance在django中解释为:

Returns the distance between the closest points on this Geometry
and the other. Units will be in those of the coordinate system of
the Geometry.

下面是测试数据:

b = [[116.27497,39.95708,2573],
[116.48103,39.96657,4292],
...
[116.13621,39.92686,528],
[116.39494,39.87986,138],
[116.389,39.8799,2151],
[116.4858,39.9361,4709]
]


创建数据:
for b in a:
AppPoint.objects.create(description=b.count,point=Point(b[0],b[1]))

输出结果:
get_point(Point(116.4,39.8),8)
{ "type": "Point", "coordinates": [ 116.4214, 39.85925 ] } 7.01270604176
{ "type": "Point", "coordinates": [ 116.33663, 39.79076 ] } 7.1289114023
{ "type": "Point", "coordinates": [ 116.43555, 39.80307 ] } 3.97213681829
{ "type": "Point", "coordinates": [ 116.42803, 39.86696 ] } 8.08069287815
{ "type": "Point", "coordinates": [ 116.41776, 39.8526 ] } 6.18016458489
{ "type": "Point", "coordinates": [ 116.41467, 39.86627 ] } 7.5557334976
{ "type": "Point", "coordinates": [ 116.37254, 39.82765 ] } 4.33799658047
{ "type": "Point", "coordinates": [ 116.36128, 39.85648 ] } 7.62292984489
{ "type": "Point", "coordinates": [ 116.41574, 39.80051 ] } 1.75308830872
{ "type": "Point", "coordinates": [ 116.40075, 39.81592 ] } 1.77417182449
{ "type": "Point", "coordinates": [ 116.45339, 39.83341 ] } 7.01111345466
{ "type": "Point", "coordinates": [ 116.39799, 39.84366 ] } 4.86535674431
{ "type": "Point", "coordinates": [ 116.38116, 39.85952 ] } 6.94973919946
{ "type": "Point", "coordinates": [ 116.3385, 39.82914 ] } 7.57577153659
{ "type": "Point", "coordinates": [ 116.3777, 39.86207 ] } 7.34200348969
{ "type": "Point", "coordinates": [ 116.39454, 39.86518 ] } 7.28121719546
{ "type": "Point", "coordinates": [ 116.41095, 39.84127 ] } 4.75311465912

##总结

    from django.contrib.gis.geos import (Polygon,Point)
import math
point = Point(130,39)
EARTH_R=6378.137
buffer = point.buffer(r*180/(math.pi*EARTH_R))
aps=AppPoint.objects.filter(point__within=buffer)

```

##新的问题
###上面这种方法得到的是没有排序的结果,目前要进行由近到远进行排序,通过
`SQRT(POW( ABS( X(Location) – X(@center)), 2) + POW(ABS(Y(Location) – Y(@center)), 2))`得到一个大致的距离因子,然后根据这个进行排序。

queryset.extra(select={‘distance_factor’: “SQRT(POWER(ABS(X(point) - “+str(x)+”),2) + POWER(ABS(Y(point) - “+str(y)+”),2))”}).order_by(‘distance_factor’)


###在线上遇到了下面的问题:

rps[0].point.distance(rps[1].point)
python: GeometryComponentFilter.cpp:35: virtual void geos::geom::GeometryComponentFilter::filter_ro(const geos::geom::Geometry*): Assertion `0’ failed.

线上直接使用distance时报错。

然后比较了一下pythondistance得到的值,其实是和`SQRT(POW( ABS( X(Location) – X(@center)), 2) + POW(ABS(Y(Location) – Y(@center)), 2))`得到的值是一样的。

(‘…..python’, 0.0071949078163964595)
(‘self’, 0.0071949078163964595)

故处些最终到终心点的距离使用了 `distance_factor` 来代替。



##注意

###[经纬度坐标系采用GCJ-2标准,对于Google,高德地图和腾讯地图可以直接使用](http://wangsheng2008love.blog.163.com/blog/static/78201689201461674727642/)
###地球坐标系 (WGS-84) 到火星坐标系 (GCJ-02) 的转换算法


####下面是通过html5获取坐标,然后转化后的当前我的位置截图
![loading](/images/project/xy_trans_position.png =x400)
####腾讯高德对其转化都有现成的实现

    var a = 6378245.0
    var ee = 0.00669342162296594323

    function out_of_china(lat,lon){
        if (lon < 72.004 || lon > 137.8347)
            return true
        if (lat < 0.8293 || lat > 55.8271)
            return true
    }
    function transformlat(x, y) {

        var result = -100.0 + 2.0 * x + 3.0 * y + 0.2 * y * y + 0.1 * x * y + 0.2 * Math.sqrt(Math.abs(x))
        result += (20.0 * Math.sin(6.0 * Math.PI * x) + 20.0 * Math.sin(2.0 * Math.PI * x)) * 2.0 / 3.0
        result += (20.0 * Math.sin(Math.PI * y) + 40.0 * Math.sin(Math.PI / 3.0 * y)) * 2.0 / 3.0
        result += (160.0 * Math.sin(Math.PI / 12.0 * y) + 320.0 * Math.sin(Math.PI / 30.0 * y)) * 2.0 / 3.0
        return result
    }
    function transformlon(x, y) {
        var result = 300.0 + x + 2.0 * y + 0.1 * x * x + 0.1 * x * y + 0.1 * Math.sqrt(Math.abs(x))
        result += (20.0 * Math.sin(6.0 * Math.PI * x) + 20.0 * Math.sin(2.0 * Math.PI * x)) * 2.0 / 3.0
        result += (20.0 * Math.sin(Math.PI * x) + 40.0 * Math.sin(Math.PI / 3.0 * x)) * 2.0 / 3.0
        result += (150.0 * Math.sin(Math.PI / 12.0 * x) + 300.0 * Math.sin(Math.PI / 30.0 * x)) * 2.0 / 3.0
        return result
    }

    function wgs2gcj(wgslat, wgslon) {
        if (out_of_china(wgslat, wgslon)) {
            return [wgslat, wgslon]
        }
        var lat = transformlat(wgslon - 105.0, wgslat - 35.0)
        var lon = transformlon(wgslon - 105.0, wgslat - 35.0)
        var rad_lat = Math.PI / 180.0 * wgslat
       var  magic = Math.sin(rad_lat)
        magic = 1 - ee * magic * magic
        var sqrt_magic = Math.sqrt(magic)
        lat = (180.0 * lat) / (Math.PI * (a * (1 - ee)) / (magic * sqrt_magic))
        lon = (180.0 * lon) / (Math.PI * a * Math.cos(rad_lat) / sqrt_magic)
        var chnlat = wgslat + lat
        var chnlon = wgslon + lon
        return [chnlat, chnlon]
    }

##参考
>0. [JAVSCRIPT Math](http://www.w3school.com.cn/jsref/jsref_obj_math.asp)

>1. [MySQL空间数据库–查询点到多点间的最短路径](http://www.javabloger.com/article/mysql-spatial-database.html)

>2. [W3 Geolocation API Specification](http://www.w3.org/TR/geolocation-API/#position_interface)

>3. [关于百度map和高德map,关于map坐标系](http://wangsheng2008love.blog.163.com/blog/static/78201689201461674727642/)

>4. [iOS 火星坐标相关整理及解决方案汇总](http://it.taocms.org/04/507.htm)